Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
Authors: Jingwei Zhang
Niklas Wetzel
Nicolai Dorka
Joschka Boedecker
Wolfram Burgard
Venue:  (2020)
Keywords: Reinforcement Learning
Exploration
Intrinsic Motivation
Sparse Rewards

Abstract: Exploration in sparse reward reinforcement learning remains an open challenge. Many state-of-the-art methods use intrinsic motivation to complement the sparse extrinsic reward signal, giving the agent more opportunities to receive feedback during exploration. Commonly these signals are added as bonus rewards, which results in a mixture policy that neither conducts exploration nor task fulfillment resolutely.
In this paper, we instead learn separate intrinsic and extrinsic task policies and schedule between these different drives to accelerate exploration and stabilize learning. Moreover, we introduce a new type of intrinsic reward denoted as successor feature control (SFC), which is general and not task-specific. It takes into account statistics over complete trajectories and thus differs from previous methods that only use local information to evaluate intrinsic motivation. We evaluate our proposed scheduled intrinsic drive (SID) agent using three different environments with pure visual inputs: VizDoom, DeepMind Lab and DeepMind Control Suite. The results show a substantially improved exploration efficiency with SFC and the hierarchical usage of the intrinsic drives. A video of our experimental results can be found at https://gofile.io/?c=HpEwTd.

Under review as a conference paper at ICLR 2020
SCHEDULED INTRINSIC DRIVE: A HIERARCHICAL TAKE ON INTRINSICALLY MOTIVATED EXPLORATION
Anonymous authors Paper under double-blind review
ABSTRACT
Exploration in sparse reward reinforcement learning remains an open challenge. Many state-of-the-ar methods use intrinsic motivation to complement the sparse extrinsic reward signal, giving the agent more opportunities to receive feedback during exploration. Commonly these signals are added as bonus rewards, which results in mixture policy that neither conducts exploration nor task fulfillment resolutely. In this paper, we instead learn separate intrinsic and extrinsic task policies and schedule between these different drives to accelerate exploration and stabilize learning. Moreover, we introduce a new type of intrinsic reward denoted successor feature control (SFC), which is general and not task-specific. It takes into account statistics over complete trajectories and thus differs from previ- ous methods that only use local information evaluate intrinsic motivation. We evaluate our proposed scheduled intrinsic drive (SID) agent using three different environments with pure visual inputs: VizDoom, DeepMind Lab and DeepMind Control Suite. The results show a substantially improved exploration efficiency with SFC and the hierarchical usage of the intrinsic drives. A video of our exper- io ce
imental results can be found at htttps://goffi.
INTRODUCTION
Reinforcement learning (RL) agents learn on evaluative feedback (reward signals) instead of in- structive feedback (ground truth labels), which takes the process of automating the development of intelligent problem-solving agents one step further (Sutton & Barto 2018). With deep networks as powerful function approximators bringing traditional RL into high-dimensional domains, deep reinforcement learning (DRL) has shown great potential (Mnih et al. 2015 2016 [Schulman et al.) 2017 Horgan et al. 2018). However, the success of DRL often relies on carefully shaped dense ex- trinsic reward signals. Although shaping extrinsic rewards can greatly support the agent in finding solutions and shortening the interaction time, designing such dense extrinsic signals often requires substantial domain knowledge, and calculating them typically requires ground truth state informa- tion, both of which is hard to obtain in the context of robots acting in the real world. When not carefully designed, the reward shape could sometimes serve as bias even distractions and could potentially hinder the discovery of optimal solutions. More importantly, learning on dense extrinsic rewards goes backwards on the progress of reducing supervision and could prevent the agent from taking full advantage of the RL framework.
In this paper, we consider terminal reward RL settings, where a signal only given when the final goal is achieved. When learning with only an extrinsic terminal reward indicating the task at hand, intelligent agents are given the opportunity to potentially discover optimal solutions even out of the scope of the well established domain knowledge.
However, in many real-world problems defining a task only by terminal reward means that the learning signal can be extremely sparse The RL agent would have no clue about what task to accomplish until it receives the terminal reward for the first time by chance. Therefore in those scenarios guided and structured exploration is crucial, which is where intrinsically-motiittee ex- ploration (Oudeyer & Kaplan 2008 Schmidhuber] 2010) has recently gained great success (Pathak) al. 2 Burda et al. 20. 2018b). Most commonly current state-of- state-of-the-ar the-art approaches, intrinsic reward is added as reward bonus to the extrinsic reward. Maximizing this combined reward signal, however, results in mixture policy that neither acts greedily with regard to extrinsic reward max-

---

Under review as a conference paper at ICLR 2020
imization nor to exploration. Furthermore, the non-stationary nature of the intrinsic signals could potentially lead to unstable learning on the combined reward. In addition. current state-of-the-art art methods have been mostly looking at local information calculated out of 1-step lookahead for the estimation of the intrinsic rewards, e.g. one step prediction error (Pathak et al. 2017), or network distillation error of the next state (Burda et al. (008b). Although those intrinsic signals can be prop- agated back to earlier states with temporal difference (TD) learning, it is not clear that this results in optimal long-term exploration. We seek to address the aforementioned issues as follows:
1.
We propose a hierarchical agent scheduled intrinsic drive (SID) that focuses one moti- vation at a time: It learns two separate policies which maximize the extrinsic and intrinsic rewards respectively. A high-level scheduler periodically selects to follow either the extrin- sic or the intrinsic policy to gather experiences. Disentangling the two policies allows the agent to faithfully conduct either pure exploration or pure extrinsic task fulfillment. More- over, scheduling (even within an episode) implicitly increases the behavior policy space exponentially, which drastically differs from previous methods where the behavior policy could only change slowly due to the incremental nature of TD learning.
2.
We introduce successor feature control (SFC), novel intrinsic reward that is based on the concept of successor features. This feature representation characterizes states through the features of all its successor states instead of looking at local information only. This implicitly makes our method temporarily extended, which enables more structured and far- sighted exploration that is crucial in exploration exploration-challengiin challenging environments.
We note that both the proposed intrinsic reward SFC and the hierarchical exploration framework SID are without any task-specific components, and can be incorporated into existing DRL methods with minimal computation overhead. We present experimental results in three sets of environments, evaluating our proposed agent in the domains of visual navigation and control from pixels, as well as its capabilities of finding optimal solutions under distraction.
RELATED WORK
Intrinsic motivation can be defined as agents conduct- ing actions purely out of the satisfaction of its internal rewarding system rather than the extrinsic rewards (Oudeyer& & Kaplan, 2008; Schmichuber] 2010)). There exist various forms of intrinsic mo- tivation and they have achieved substantial improvement in guiding exploration for DRI., in tasks where extrinsic signals are sparse or missing altogether.
Intrinsic Motivation and Auxiliary Tasks
Pathak et al. 2017) proposed to evaluate curiosity, one of the most widely used kinds of intrinsic motivation, with the 1-step prediction error of the features of the next state made by a forward dynamics model. Their ICM module has been shown to work well in visual domains including first-person view navigation. Since ICM is potentially susceptible to stochastic transitions (Burda) et al. 2018a), Burda et al. (2018b) propose as a reward bonus the error of predicting the features of the current state output by a randomly initialized fixed embedding network. The value function is decomposed for extrinsic and intrinsic reward, but different to single mixture policy is learned. Another form of curiosity, learning progress or the change in the prediction error, has been connected to count-based exploration via a pseudo-count (Bellemare et al. 2016 Ostrovski et al. 2017) and has also been used as a reward bonus. Savinov et al. (2018) propose to train reachability network, which gives out reward based on whether the current state is reachable within a certain amount of steps from any state in the current episode. Similar to our proposed SFC, their intrinsic motivation related to choosing states that could lead to novel trajectories. However, we use two different distance metrics, theirs is explicitly learned to be proportional to the time step differences while ours is based on successor features which measures two states by the difference of the average feature activations of future trajectories. Moreover, their method rewards states with high distance to the states in the current episode while our method rewards states with high distance to the states also from past trajectories, the successor features are trained from samples of the replay buffer.
Auxiliary tasks have been proposed for learning more representative and distinguishable features. Mirowski et al. (2016) add depth prediction and loop closure prediction auxiliary tasks for leamn ing the features. Jaderberg et al. (2016) learn separate policies for maximizing pixel changes (pixel control) and activating units of specific hidden layer (feature control). However, their proposed
2

---

Under review as a conference paper at ICLR 2020
UNREAL agent never follows those auxiliary policies as they are only used to learn more suitable features for the main extrinsic task.
Hierarchical RL Various HRL approaches have been proposed (Kulkarni et al. 2016a Bacon et al., 2017; Vezhnevets al., 2017; Krishnan et al. 2017). In the context of intrinsic motivation, feature control (Jaderberg et al., 2016) has been adopted into a hierarchical setting (Dilokthanakul al. 2017). in which options are constructed for altering given features. However, they report that a flat policy trained on the intrinsic bonus achieves similar performance to the hierarchical agent.
Our hierarchical design is perhaps inspired mostly by the work of Riedmilller al. (2018). Un- like other HRL approaches that try to learn set of options (Sutton et al.) 1999) to construct the optimal policy. their proposed SAC agent aims to learn one flat policy that maximizes the extrinsic reward. While SAC schedules between following the extrinsic task and set of pre- auxiliary tasks such as maximizing touch sensor readings or translation velocity, in this paper we investigate scheduling between the extrinsic task and intrinsic motivation that is general and not task-specific. A concurrent work along this line is presented by [eyer et al. (2019).
Successor Representation The successor representation (SR) was first introduced to improve gen- eralization in TD learning (Dayan 1993). While previous works extended SR the deep setting for better generalized navigation and control algorithms across similar environments and changing goals (Kulkarni et al. 2016b] Barreto et al. 2017 /Zhang al. 2017). focus on its temporarily
extended property to accelerate exploration.
SR has also been investigated under the options framework. Machado al. (2017); Tomar* et al. (2019) evaluate successor features with random policies to discover bottlenecks or landmarks based on the clustering of such features. Options are then learned to navigate those sub-goals. However, it remained unclear if the options framework would help in sparse exploration setups.
When using SR measure the intrinsic motivation, the most relevant work ours is that of Machado et al. (2018). They also design a task-independent intrinsic reward based on SR, how- ever they rely on the concept of count-based exploration and propose reward bonus, that vastly differs from ours. Their bonus is inverse proportional to the norm of the SR while our formulation section.
rewards change the SR of two successive states. We will present our proposed method in the next
METHODS
We use the RL framework for learning and maaiiinn-making under uncertainty. is formalized by Markov decision processes (MDPs) defined by the tuple p,, At time step & the agent samples an action a E A according to policy π (-1s), which depends on its current state E S. The agent receives scalar reward r R and transits to the next state s' S. The distribution of the corresponding state, action and reward process (S,,A, Re+1) is determined by the distribution of the initial state Sg. the transition operator p and the policy т. The goal of the agent is find policy that maximizes the expectation of the sum of discounted rewards Σ 7 Re+k+1- We seek to speed up learning in sparse reward RL, where the reward signal is uninformative for almost all transitions. We set the focus on terminal reward scenarios, where the agent only receives a single reward of +1 for successfully accomplishing the task and 0 otherwise.
We will first introduce our proposed intrinsic reward successor feature control (SFC) 3.1 3.2 then present our proposed hierachical framework for accelerating intrinsically motivated exploration, which we denote as scheduled intrinsic drive (SID) (Sec 3.4)
3.1 SUCCESSOR DISTANCE METRIC
In order encode long-term statistics into the design of intrinsic rewards for far-sighted exploration, we build on the formulation of successor represention (SR), which introduces a temporally extended view of the states. Dayan (1993) introduced the idea of representing state by the occupancies of all other states from a process starting in s following a fixed policy म, where the occupancies denote the average number of time steps the state process stays in each state per episode. Successor
3

---

Under review as a conference paper at ICLR 2020
(a) Successor Distance
(b) SFC Rewards
(c) Random Exploration (d) Exploration with SFC
The agent starts at the red cross and transitions explore the four rooms when no extrinsic reward is provided. In a) each state is annotated by its SD (Eq(3) to the starting state and b) shows for each state the highest possible SFC reward (Eq 4 for one-step transition from it. Here the successor features are learned using a random walk. c) and d) show a comparison between visitation counts of each state from a random agent and an agent that uses the SFC rewards for control via Q-learning- In the latter case the successor features are learned from scratch via TD.
Figure 1:
The four-room domain (Sutton et al. 1999).
an adjacent state each time step. The goal
In this environment, the agent receives high rewards for crossing bottleneck states, when the SF are learned beforchand, using random policy. But even when the SF are learned during exploration, bottleneck states are still visited disproportionately high. Furthermore the intrinsic reward greatly improves exploration compared random agent. For implementation details Appendix, D.4
features (SF) (Kulkarni et al. 2016b Barreto et al. 2017 extend the concept to an arbitrary feature embedding Φ R For a fixed policy π and embedding & the SF defined by the |m|- dimensional vector
(x Ex
Σ r (')( f=
(1)
Analogously, the SF represent the average discounted feature activations, when starting in s and following æ. They can be learned by temporal difference (TD) updates
Sr (1)) + Un (S,) + + ((S) + (V,4)(S411) Wn.o(S))..
(2)
SF have several interesting properties which make them appealing as a basis for an intrinsic reward signal: 1) They can be learned even in the absence of extrinsic rewards and without learning tran- sition model and therefore combine advantages of model-based and model-free RL (Stachenfeld et al. 2014). 2) They can be learned via computationally efficient TD. 3) They capture the expected feature activations for complete episodes. Therefore they contain information even of spatially and temporarily distant states which might help for effective far sightd exploration. Given the discus- sion, we introduce the successor distance (SD) metric that measures the distance between states by the similarity of their SF
(3) Fig a) shows an example of the successor distance metric in the tabular case. There the SD roughly correlates to the length of the shortest path between the states. Using this metric to evaluate the intrinsic motivation, one choice could be to use the SD to fixed anchor state the intrinsic reward, which depends heavily on the anchor position. Even when a sensible choice for the anchor can be found, e.g. the initial state of episode, the SDs of distant states from the anchor assimilate.
dr,o(8, = (lV (8')l2
For pair of states with a fixed spatial distance, their SD is higher when they are located in different rooms and the SD increases substantially when crossing rooms. Therefore the metric might capture the connectivity of the underlying state space.
3.2 SUCCESSOR FEATURE CONTROL
This observation motivates us to define the intrinsic reward successor feature control (SFC) as the squared SD of pair of consecutive states
R+ 11r,o(Se+ (S))(3
(4)
4

---

Under review as a conference paper at ICLR 2020
A high SFC reward indicates big change in the future feature activations when π is followed. We argue this big change is a strong indicator of bottleneck states, since in bottlenecks a minor change in the action selection can lead to vastly different trajectory being taken. Fig[Tp) shows that those highly rewarding states under SFC and the true bottlenecks agree, which can be very valuable for exploration (Lehnert et al. 2018).
3.3 SCHEDULED INTRINSIC DRIVE
The classical way of adding the intrinsic reward to the extrinsic reward has several drawbacks. First, the final policy not trained to maximize the actual objective but a mixed version. Second, the intrinsic reward signal is usually changing over time. Including this non-stationary signal in the overall reward can make learning of the actual task unstable. Furthermore, the performance is often extremely sensitive to the scaling of the intrinsic reward relative to the extrinsic and hence it has to be tuned very carefully for every environment.
To overcome these issues we propose scheduled intrinsic drive (SID), which learns two separate policies, one for each reward signal. During each episode the scheduler samples several times which of the two policies to follow for the next time steps. Each policy is trained off off-policy from all of the transitions irrespective of which policy collected the data.
As SID does not add the two reward signals no scaling parameter is needed. A policy is learned that exclusively maximizes extrinsic reward and hence neither the final policy nor the learning process is disturbed by the intrinsic reward. At the same time exploration ensured as there is experience collected by the policy that learns from the intrinsic reward. Furthermore, scheduling can help exploration as each policy is acted on for extended time interval, allowing long-term exploration instead of local exploration. Besides that the agent is less susceptible to always go to a nearby small reward instead of looking for other larger rewards that maybe further away. mixture policy might be attracted to the small reward while with SID the exploration policy is followed for several timesteps which can bring the agent to new states with larger rewards that it did not know of before.
We investigated several types of bigh-level schedulers. however, none of them consistently outper- forms a random one. We present possible explanations why random scheduler already performs well and present them in Appendix F along with the different scheduler choices we tested.
3.4 ALGORITHM IMPLEMENTATION
Our proposed method can combined with an any approach that allows off-policy learning. This section describes instantiation of the SID framework when using Ape-X DQN as basic off- policy DRL algorithm Horgan et al. (2018) with SFC as the intrinsic reward, which we used for all experiments. We depict this algorithm instance in Appendix Figure and more details are provided in Appendix C The algorithm is composed of:
A Q-Net (0 OF, 01): Contains shared embedding 0, and two Q-value output heads On: (extrinsic) and 0, (intrinsic).
A SF-Net {0p. ] Contains an embedding 0g and a successor feature head 0g.. Os is ini- tialized randomly and kept fixed during training. The output of SF-Net is used to calculate the SFC intrinsic reward (Eq[4) The SF-net is trained with the samples of the replay buffer, which contains the experience generated by the behavior policy.
A high-level scheduler: Instantiated in each actor, selects which policy to follow (extrin- sic or intrinsic) after fixed number of environment steps (max episode length/M). The scheduler randomly picks one of the tasks with equal probability.
parallel actors (N = 8): Each actor instantiates its own copy of the environment, peri- odically copies the latest model from the learner. We learn from K-step targets (K 5),
so each actor at each environment step stores (s--K Gt- K Σk=1 ** A - Kkk 8t) into shared replay buffer. Each actor will act according to either the extrinsic or the intrinsic policy based on the current task selected by its scheduler.
A learner: Learns the Q-Net (9g and 0 are learned with the extrinsie and intrinsic reward re- spectively) and the SF-Net from samples (Eq(2) from the same shared replay buffer, which contains all experiences collected from following different policies.
5

---

Under review as a conference paper at ICLR 2020
(a) MyWayHome
(b) Fscape
111
(c) Corridor.
(d) Exit.
(e) Wing.
(I) Goal.
Figure 2: VizDoom environments we evaluated on. 2a and 2b show the top-down views of My- WayHome and FlytrapEscape with the same downscaling ratio, with red dots marking the starting locations, green dots indicating the goal locations; 2c and 2d show exemplary first-person views captured from the marked poses (blue dots with arrows) from those two maps respectively.
EXPERIMENTS
We evaluate our proposed intrinsic reward SFC and the hierarchical framework of intrinsic motiva- tion SID in three sets of simulated environments: VizDoom (Kempka et al. 2016). DeepMind Lab (Beattie et al. 2016) and Deep.Mind Control Suite TTssa et al. (218). Throughout all experiments, agents receive input only pixels with additional domain knowledge
task specifie in- formation. We mainly compare the following agent configurations: M: Ape-X DQN with 8 actors, train with only the extrinsic main task reward: ICM: train a single policy with the ICM reward bonus PPahhkk et al. 2017); RND: train single policy with the RND reward bonus (Burda et al. 201 8b); Ours: with our proposed SID framework, schedule between following the extrinsic main task policy and the intrinsic policy trained with our proposed SFC reward.
We carried out an ablation study, where we compare the performance of an agent with intrinsic and extrinsic reward summed up, to the corresponding SID agent for each intrinsic reward type (ICM, RND, SFC). We present the plots and discussions Section 4.4 Appendix
For the intrinsic reward normalization and the scaling for the extrinsic and intrinsic rewards we do a parameter sweep for each environment (Appendix C...n and choose the best setting for each agent. We notice that our scheduling agent is much less sensitive different scalings than agents with added reward bonus. Since our proposed SID setup requires an off-policy algorithm to learn from experiences generated by following different policies, implement all the agents under the Ape-X X DQN framework Horgan et al.) (2018). After parameter sweep we set the number of scheduled tasks per episode to M 8 for agent in all experiments, meaning each episode is divided into up to 8 sub-episodes, and for each of which either the extrinsic or the intrinsic policy is sampled as the behavior policy. Appendix C and D contain additional information about experimental setups and model training details.
4.1 VIZDOOM: SPARSE NAVIGATION
We start by verifying our implementation of the baseline algorithms in "DoomMyWaHHonee which was previously used several ofte--of-the-art intrinsic motivation papers (Pathak et al. 2017 Savi- nov et al., 2018). The agent needs navigate based only first-person view visual inputs through rooms connected by corridors (Fig each with a distinct texture (Fig2c). The experimental results shown in Fig 3 (left). Since basic RL algorithm is doing off-policy learning, it has

---

Under review as a conference paper at ICLR 2020
1.0
Extrnsic rewordd per cpiede
Our ICM
Exrinii reward per sprodd
0.8
RND
0.0
0.0
0.4
0.4
0.2
0%
%
0.5
1.5
2.0
2.5
Number of perameter upclatees
Number of peremeter updates
Figure 3: Extrinsic rewards per episode obtained in MyWayHome (left) and FlytrapEscape (right). Each plot shows the mean over 5 non-tuned random seeds. Figures showing the learning curves for each run can be found in the Appendix in Figure and 13
relatively decent random exploration capabilities. We see that the M agent is able solve the task sometimes without any intrinsically generated motivations, but that all intrinsic motivation types help to solve the task more reliably and speed up the learning. Our method solve the task the fastest, but also ICM and RND learn to reach the goal reliably and efficiently.
We wanted to test the agents on a more difficult VizDoom map where structured exploration would be of vital importance. We thus designed new map which scales up the navigation task of MyWay- Home. Inspired by how flytraps catch insects, we design the layout of the rooms in a geometrically challenging way that escaping from one room to the next with random actions is extremely unlikely. We show the layout of My Wayee (Fig 2a) and FlytrapEscape (Fgg[b with the same downscal- ing ratio. The maze consists of 4 rooms separated by V-shaped walls pointing inwards the rooms. The small exits of each room is located at the junction of the V-shape, which is extremely difficult to maneuver into without a sequence of precise movements. As in the My MyWayHome task, in each episode, the agent starts from the red dot shown in 2bg with a random orientation. An episode terminates if the final goal is reached and the agent will receive reward of +1, or if maximum episode steps of 10,000 (2 (2100 for My WayHome) is reached. The task is to escape the fourth room.
The experimental results on FlytrapEscape are shown in Fig 3 Neither M nor RND manages to learn any useful policies. ICM solves the task in sometimes, while we can clearly observe that our method efficiently explores the map and reliably learns how to navigate the goal. We visualize the learned successor features in Appendix and its evolution over time is shown in the video
4.2 DEEPMIND LAB: EXPLORATION UNDER DISTRACTION
In the second experiment, we set out to evaluate the agents would be able to reliably collect the faraway big reward in the presence of small nearby distractive rewards. For this experiment use the 3D visual navigation simulator of DeepMind Lab (Beattie et al.] 2016). We constructed challenging level "AppleDDisrractions" (Fig (Obg with maximum episode length of 1350. In this level, the agent starts in the middle of the map (blue square) and can follow either of the two corridors. Each corridor has multiple sections and each section consists of two dead-ends and an entry to next section. Each section has different randomly generated floor and wall textures. One of the corridors (left) gives a small reward of 0.05 for each apple collected, while the other one (right) contains a single big reward of at the end of its last section. The optimal policy would be to go for the single faraway big reward. But since the small apple rewards are much closer to the spawning location of the agent, the challenge here is to still explore other areas sufficiently often so that the optimal solution could be recovered.
The results are presented in Fig 4 (left). Ours received on average the highest rewards and is the only method that learns to navigate to the large reward in every run. The baseline methods get eas- ily distracted by the small short-term rewards and do not reliably learn to navigate away from the distractions. With a separate policy for intrinsic motivation the agent can for some time interval completely "forget" about the extrinsic reward and purely explore, since it does not get distracted by the easily reachable apple rewards and can efficiently learn to explore the whole map. In the mean- while the extrinsic policy can simultaneously learn from the new experiences and might learn about
7

---

Under review as a conference paper at ICLR 2020
v
1.0
Dves KM BDD
rewerd per sptodd
0.8
Exerac Berrscrewerd
0.4
0.4
Extrnsic
8..0 0.3
0.6 0.9
Number parsmeter updates
Number of peremneeer updattes
Figure 4: Extrinsic rewards per episode obtained in AppleDistractions (left) and Cartpole (right). Each plot shows the mean over 5 non-tuned random seeds. Left: Each agent is evaluated on the same 5 sets of random floor and wall textures, with 5 non-tuned environment seeds. In the ablation study (Appendix (A) the SID variant outperforms the reward bonus variant of each of the 3 types of intrinsic rewards. Right: Ours also outperforms all baseline agents the very different domain of classic control from pixels, which shows the general applicability of our proposed agent. Figures showing the learning curves for each run can found in the Appendix in Figure T5 and 16
the final goal discovered by the exploration policy. This highlights a big advantage of scheduling over bonus rewards, that it reduces the probability of converging to bad local optimums. In Section 4.4 we further showed that SID generally applicable and also helps ICM and RND this task.
4.3
DEEPMIND CONTROL SUITE: CLASSIC CONTROL FROM PIXELS
To show that our methods can be used in domains other than first-person visual navigation, we evaluate the classic control task "cartpole: swingup sparse" (DeepMind Control Suite Tassa et al. (2018), using third-person view images as inputs [Fig[T].. The pole starts pointing down and the agent receives single terminal reward of -1 for swinging up the unactuated pole using only horizontal forces on the cart. Additional details are presented in Appendix D.3 The results are shown in 4 (right). Compared the previous tasks, this task is easy enough to be solved without intrinsic motivation, but we can see also that all intrinsic motivation methods significantly reduce the interaction time. Ours still outperforms other agents even in the absence of clear bottlenecks which shows its general applicability, but since the task is relatively less challenging for exploration, the performance gain not as substantial as the previous experiments.
4.4 ABLATION STUDY
Further, we conducted an ablation study on AppleDistractions We denote with "M+SFC", "M+RND", "M+ICM" the agents with one policy where the respective intrinsic reward is added to the extrinsic one. With "SID(M,FFC)", "SID(M,RND)", "SID(M,CMM)" the agents are named that have two policies, one for the respective intrinsic and one for the extrinsic reward and use SID
1.0
(M,SFC)
Extrinsic rewards per episode
M+SFC
0.8
SID (MUCCM)
SID (M.NND)
0.6
M+RND
0.4
0.2
0.0
0.0
0.3
0.6
0.9
1.2
1.5
Number of parameter updates
Figure 5: Ablation study results for AppleDistractions. Each plot shows the mean over non-tuned random seeds.
8

---

Under review as a conference paper at ICLR 2020
to schedule between them. We note that the "SID(MICM)) agent corresponds the "Ours" agent from the previous experiments. The results are presented Figure 5 Our SID(M, SFC) agent received on average the highest rewards. Furthermore, and SFC to find the goal and not settle for the small rewards, and SID also helps improve the per- formance of RND. The respective reward bonus counterparts of the three SID agents attracted to the small nearby rewards, This behavior is expected: By scheduling, the intrinsic policy of the SID agent assigned with its own interaction time with the environment, during which could completely "forget" about the extrinsic rewards. The agent then has a much higher probability of discovering the faraway big reward, thus escaping the distractions of the nearby small rewards. Once the intrinsic policy collects these experiences of the big reward, the extrinsic policy can im- mediately learn from those since both policies share the same replay buffer. Ablations for the other environments are reported in Appendix
that scheduling helped both ICM
more
5 CONCLUSION
In this paper, we investigate an alternative way of utilizing intrinsic motivation for exploration in DRL. We propose hierarchical agent SID that schedules between following extrinsic and intrinsic drives. Moreover, intrinsic motivation based on longer time horizons. We conduct experiments in three sets of envi- ronments and show that both our contributions SID and SFC help greatl in improving exploration efficiency.
propose
type of intrinsic reward SFC that is general and evaluates the
We consider many possible research directions that could stem from this work, including design- ing more efficient scheduling strategies, incorporating several intrinsic drives (that are possibly or- thogonal and complementary) instead of only into SID, testing framework in other control domains such as manipulation, combining the successor representation with learned feature repre- sentations and extending our evaluation onto real robotics systems.
REFERENCES
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, pp. 1726- 1734, 2017.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances neural information processing systems, pp.
2017.
Charles Beattie, Joel 7. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.0380) 2016.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-14799 2016.
Lucas Beyer, Damien Vincent, Olivier Teboul, Sylvain Gelly, Matthieu Geist, and Olivier Pietquin. Mulex: Disentangling exploitation from exploration in deep rl. arXiv preprint arXiv: 190.008,,,,0 2019.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv: 04355, 2018
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 18100 120.1229944. 2018b.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5(4):613-624, 1993.
Nat Dilokthanakul, Christos Kaplanis, Nick Pawlowski. and Murray Shanahan. Feature control as
intrinsic motivation for hierarchical reinforcement learning. arXiv preprint arXiv:
2017.

---

Under review as a conference paper at ICLR 2020
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint wr.iv:1802.01.5000 2018.
Dan Horgan, John Quan, David Budden, Gabriel Marth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv: 00930, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint
2015.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv: 161 1.05397, 2016.
Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaškowski. Viz- doom: A doom-based ai research platform for visual reinforcement learning. In Computational Intelligence and Games (CIG), 2016 IEEE Conference on, pp. 1-8. IEEE, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. Ddco: options for robot learning from demonstrations. In Conference on Robot Learning, pp. 418-437, 2017.
Discovery of deep continuous
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675-3683, 2016a.
Tejas Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel Gershman. Deep successor reinforcement learning. arXiv preprint 1606.
2016b.
Lucas Lehnert, Romain Laroche, and Harm van Seijen. On value function representation of long horizon problems. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint 17Xiv:1710.11089, 2017.
Marlos C Machado, Mare G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. arXiv preprint arXiv: 1807. 1162,, 2018.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukeuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.03675 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 51897540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp.
2016.
Georg Ostrovski, Mare Bellemare, Aaron van den Oord, and Rémi Munos. Count-based explo- ration with neural density models. arXiv preprint 1703:01003 2017.
Pierre Yves Oudeyer and Frederic Kaplan. How can we define intrinsic motivation? In Proceedings of the 8th International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems, Lund University Cognitive Studies, Lund: LUCS, Brighton. Lund University Cognitive Studies, Lund: LUCS, Brighton, 2008
10

---

Under review as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), 2017.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch. arXiv preprint httt:/880211066/1 201000066,2210 2018.
Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys, Timo- thy Lillicrap. and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint arXiv:1810.022274, 2018.
Jürgen Schmidhuber. Formal theory creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):230-24,, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1X07.06377003477 2017.
Kimberly L Stachenfeld, Matthew Botvinick, and Samuel Gershman. Design principles of the hippocampal cognitive map. In Advances in neural information processing systems, pp. 2528- 2536, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard $ Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-
211. 1999.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1tt.:10011..0900 2018
Manan Tomar* Rahul Ramesh*, and Balaraman Ravindran. Successor options An option discov- cry algorithm for reinforcement learning. 2019. URLAtttss://ppenreviie net forum um?
1Id=Byx27RR5PP
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv:1703.011 2017.
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep rein- forcement learning with successor features for navigation across similar environments. In Intel- ligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on, pp. 2371-2378. IEEE, 2017.

---

Under review as a conference paper at ICLR 2020
1.0
(M.57C)
Eshrnsk: meants per ephecde
0.8
(M.(CM)
0.0
M+|CM
SID (M (M.RND)
0.0
M+RND
0.4
0.4
SID (MLSE
Extriicc
510 (MLIC))
0.2
M--CH
5I0 (NLRND)
AND
08.0
0.5
1.0
2.0
2.5
0.0
Number parameter updates
Numbbr of parameter updattes
Figure 6: Ablation study results for FlytrapEscape (left) and Cartpole (right).
A APPENDIX: FURTHER ABLATION STUDIES
We have conducted ablation studies for all the three sets of environments investigate the influence of scheduling on our proposed method, whether other reward types can benefit from scheduling too, and whether environment specific differences exist.
We compare the performance of the following agent configurations:
Three reward bonus agents M+ICM, M+RND, M4SFC:
The agent receives the intrinsic reward of ICM (Pathak et al. 2017). RND (Burda et al, 2018b) or our proposed SFC respectively as added bonus to the extrinsic main task reward and trains a mixture policy on this combined reward signal. We note that the M+ICM and M+RND agent in this section corresponds to the ICM and RND agent in all other sections respectively.
Three SID agents SID (M, ICM), SID (M, RND), SID (M, SFC):
The agent schedules between following the extrinsic main task policy and the intrinsic policy trained with the ICM, RND or our proposed SFC reward respectively.
We note that the SID (M, SFC) agent in this section corresponds to the Ours agent in all other sections.
The results on AppleDistractions were shown in in Section 4.4 the main paper. In Fig 6 (left), we present the ablation study results for FlytrapEscape. The agents with the ICM component per- form poorly. Only 1 run of M+ICM learned to navigate to the goal, while the scheduling agent SID(MICM) did not solve the task even once. But for the two SFC agents, the scheduling greatly improves the performance. Although the reward bonus agent M+SFC was not successful in every run, the SID(M,SFC) agent solved the FlytrapEscapp in 5 out of 5 runs. We hypothesize the reason for the superior performance of SID(M,SFC) compared to M+SFC could be the following: Before seeing the final goal for the first time, the M+SFC agent is essentially learning purely on the SFC reward, which is equivalent to the intrinsic policy of the scheduling SIFC agent. Since SFC might preferably go to bottleneck states as the difference between the SF of the two neighboring states are expected be relatively larger for those states. Since the extrinsic policy doing random exploration before receiving any reward signal, it could be a good candidate to explore the next new room from the current bottleneck state onwards. Then the SFs of the new room will be learned when it is being explored, which would then guide the agent to the next bottleneck regions. Thus the SID(M,SFC) agent could efficiently explore from bottleneck bottleneck, while the M+SFC agent could not be able to benefit from the two different behaviors under the extrinsic and intrinsic re- wards and could oscillate around bottleneck states. On the other hand, scheduling did not help ICM or RND. A reason could be that ICM or RND is not especially attracted by bottleneck states it does not help exploration if the agent spends half of the time acting randomly as the extrinsic policy had no reward yet to learn from. Also since the FlytrapEscape environment extremely exploration- challenging, the temporally extended view of our proposed SFC might of vital importance to guide efficient exploration.
In (right), we present the ablation study results for Cartpole. We can observe that SID helps to improve the performance of both ICM and RND. As for SFC, although the reward bonus agent
12

---

Under review as a conference paper at ICLR 2020
0.8
16
0.6
Extrinsic reward
0.4
0.0
4
Number of parameter updates
Figure 7: Results on "DoomMyyaaHome' for different times of scheduling per episode in SID with SFC as intrinsic reward.
learns bit faster than the SID agent, we note that actually all the three SID agent converge to more stable policies, while the reward bonus agents tend oscillate around the optimal return.
B APPENDIX: NUMBER OF SWITCHES PER EPISODE
experiments of the main paper with SID the scheduler chooses times per episode which policy to follow until new policy is chosen. We conducted a further experiment to examine how different numbers of switches per episode affects the performance of the agent. We carried out the experiment on "DoomMy WayHome" with agent using SID with SFC for the intrinsic reward. The results show in 7
C
APPENDIX: IMPLEMENTATION DETAILS
This section describes implementation details and design choices. The backbone of our algorithm implementation presented in Section 3.4 and visualized in Fig.
C.I
APE-X DQN
Since our algorithm requires off-policy learning strategy, and in consideration for faster learning and less computation overhead, we use the state-of-the-ar off-policy algorithm Ape-X DQN Horgan al. (2088) with the K ststep target (K 5) for bootstraping without off off-policy correction
k=
K Retk + max ((s--K arg max
yt
d'
A=1
where 0 denotes the target network parameters.
We chose the number of actors the be the highest the hardware supported, which was 8. To adapt the settings from the 360 actors in the Ape-X DQN to setting of N 8 actors, set fixed & for each actor E (1....,8) as
@ 300
360-1
(5)
where a = 7 and e = 0.4 are set as in the original work.
C.2 PRIORITIZED EXPERIENCE REPLAY
For computational efficiency, we implement our own version of the prioritized experience replay. We split the replay buffer into two, with size of 40, 000 and 10, 000. Every transition is pushed to the first one, while the second one only transitions are pushed on which a very large TD-error computed. We store running estimate of the mean and the standard deviation of the TD-error
13

---

Under review as a conference paper at ICLR 2020
env _
env N
scheduler periodically
scheduler N periodically samples behavior policy
experience tuple
experience tuple
samples behavior policy
QF Q1
actor 1
actor_ _
Q
experience tuple
experience tuple
replay buffer
samples minibatches {s, rE,
learner
train w/ {8, TE, s'}
train {s, TI, s'
train w/ {s. s
QE
Q1
gives SFC
periodically copied to actors
Figure 8: Flow diagram of the algorithm implementation (Sec 3.4 3.4)
and if for transition the error is larger than the mean plus two times the standard deviation, the transition is pushed. In the learner a batch of size 128 consists of 96 transitions drawn from the normal replay buffer and 32 are drawn from the one that stores transition with high TD-error, which as result have relatively seen a higher chance of being picked.
C.3 SUCCESSOR FEATURE LEARNING
We note that previous works for learning the deep SF have included an auxiliary task of reconstruc- tion on the features & Kulkarni et al. (2016a); Zhang et al.) (2017), while in this work we investigate learning v without this extra reconstruction stream. Instead of adapting the features 6 while learning the successor features v, we the randomly initialized d. This design follows the intuition that since SF (2)) estimates the expectation of features (d) under the transition dynamics and the policy being followed, more stable learning of the SF could achieved if the features are kept fixed.
The SF are learned from the same replay buffer for training the Q-Net. Since our base algorithm is K Ktstep Ape-X, and we follow the memory efficient implementation of the replay buffer as suggested in Ape-X, we only have access K K-step experience tuples (K 5) for learning the SF. Therefore we calculate the intrinsic reward by applying the canonical extension of the SFC reward formulation 4) K-step transitions
Rsfe RL+K
|| (St+k) Wr. (5:)1
14

---

Under review as a conference paper at ICLR 2020
E
# #actions
4x4 rilters
64 filters
Q-Net
filters
512
11
512
#actions
θ Ø
0y:
8x8 filters 5-4
64 4x4 fitters 5-2
64 3x3 filters s-1
SF-Net
512
512*
512
512
Figure 9: Model architecture the SID (M, SFC) agent. Components with color yellow are randomly intialized and not trained during learning.
The behaviour policy æ associated with the SF is not given explicitly, but since the SF are learned from the replay buffer via TD learning, mixture of current and past behaviour policies from all actors.
C.4 REWARD NORMALIZATION
Most network parameters are shared for estimating the expected discounted return of the intrinsic and extrinsic rewards. The scale of the rewards has a big influence on the scale of the gradients for the network parameters. Hence, it is important that the rewards are roughly on the same scale, otherwise effectively different learning rates are applied. The loss of the network comes from the regression the Q-values, which approximate the expected return. So our normalization method aims to bring the discounted return both tasks into the same range. To so we first normalize the intrinsic rewards by dividing them by running estimate of their standard deviation. We also keep a running estimate of the mean of this normalized reward and denote r'y Since every time step an intrinsic reward is received estimate the discounted return via the geometric series. We
scale the extrinsic task reward that is always in {0, 13 with 7) 71 where "/r is the discount rate for the intrinsic reward. Furthermore, 17 is hyperparameter which takes into account that for Q-values from states more distant the goal the reward discounted with the discount rate for the extrinsic reward depending on how far away that state is. In our experiments we set 3.
We did the same search for hyperparameters and normalization technique for all algorithms that include an intrinsic reward and found out that the procedure above works best for all of them. The algorithms were evaluated on the FlytrapEscape. For rj we tried the values in {0.3, 1, 10). We also tried to not normalize the rewards and just scale the intrinsic reward. To scale the intrinsic reward we tried the values (0.001, 0.01, 0.1, 1). However, we found that as the scale of the intrinsic over the whole training process this approach does not work well. We also
rewards is not the tried to normalize the intrinsic rewards by dividing it by a running estimate of standard deviation and then scale this quantity with value in (0.01, 0.1, 1).
C.5 MODEL ARCHITECTURE
We the model architecture depicted in Fig. acroos all 3 sets of experiments.
ReLU activations are added after every layer except for the last layers of each dashed blocks in the above figure. For the experiments with the ICM (Pathak et al. (017). we added BatchNorm (loffe & SZzeedyy 2015) before activation for the embedding of the ICM module following the original code released by the authors. Code implemented in pytorch (Paszke et al. 2017)
15

---

Under review as a conference paper at ICLR 2020
L LIITH
100 100
(b) Top-down view
(a) Entry:
(c) Goal.
Figure 10: Top-down view and exemplary first-person view observations captured in the AppleDis- tractions environment.
D APPENDIX: TRAINING DETAILS
We use batch size of 128 for all experiments the Adam optimizer (Kingma & Ba 2014) with learning rate of le - 4.
For all experiments we used stack of consecutive, preprocessed observations states.
For the first-person view experiments in VizDoom and DeepMind Lab, we use an action repetition of 4, while for the classic control experiment did not apply action repetition. In the text, we only refer to the actual environment steps (e.g. before divided by 4).
D.1 ENVIRONMENT SETTINGS: VIZDOOM
The VizDoom environment produces 320 x 240 RGB images as observations. In a preprocessing step, downscaled the images 84 x 84 pixels and converted them grayscale.
For FlytrapEscape, we adopted the action space settings from the MyWayHome task. The ac- tion space was given by the following 5 actions: TURN LEFT, TURN RIGHT, MOVE FORWARD, MOVE_LEFT, MOGH_RIGHT
D.2 ENVIRONMENT SETTINGS: DEEPMIND LAB
We setup the DmLab environment to produce 84 x 84 RGB images as observations. In Fig 10 we show examplary observations of AppleDistractions. We preprocessed the images by converting the observations to grayscale.
For given enviroment seed, textures for each segment of the maze are generated at random.
We used the predefined DmLab actions from Espeholt et al. (2018). The action space was given by the following 8 actions (no shooting setting): Forward, Backward, Strafe Left. Strafe Right, Look Left, Look Right, Forward+ Left, Forward+Look Right.
D.3 ENVIRONMENT SETTINGS: DEEPMIND CONTROL SUITE
We conducted the experiments for the classic control task on the 'Cart-pole' domain with the 'swingup-spae sparse" task provided by the DeepMind Control Suite. Since our agents needs discrete action space and the control suite only provides continuous action spaces, we discretized the single action dimension. The set of actions was (-0.5, -0.25, 0, 0.25, 0.5). We configured the environment to produce 84x84 RGB pixel-only observations from the 1st camera, which is the only predefined camera that shows the full cart and pole all times. We further convert the images to grey-scale and stack four consecutive frames as input to our network. The episode length was 200 environment steps.
16

---

Under review as a conference paper at ICLR 2020
(a) Start
(b) Swingup
(c)
Figure 11: Exemplary observations captured in the Cartpole environment.
D.4 ENVIRONMENT SETTINGS: FOUR-ROOM
In the four-room domain, the agent can transition to directly connected states using the four actions 'up'. "down', 'left' and 'right'. For Fig b)) the successor features were calculated analytically via the formula v factor was set to y = 0.95.
PP 1, where P' denotes the one-step transition matrix. The SF discount
For d)) the agents performed 10000 episodes with 30 steps each. These short episodes ensure that exploration remains challenging, even in a relatively small environment. In d). the learning rate for the SF well as the Q-table was set to 0.05. To prevent optimistic initialization effects the Q-table was initialized to O.
D.5 INFRASTRUCTURE
To generate results we used two machines that run Ubuntu 16.04. Each machine has 4 GeForce Titan X (Pascal) GPUs. On one machine we run experiments parallel, each experiment on separate GPU,
E APPENDIX: SUCCESSOR DISTANCE VISUALIZATION
-30
Projection of the SFs. For the purpose of visualization we discretized the map into 85 x 330 grids and position the trained agent SID(M,SFC) each grid, then computed the successor features & for that location for each of the orientations (0", 90" 180 270%), which resulted in x 512 matrix. then calculated 12-difference of this matrix with a 512 vector containing the successor features of the starting position with the 4 different orientations. Shown in log-scale.
Figure 12:
As an additional evaluation, we visualize the SF of an Ours agent (i.c. trained with SID which schedules between the extrinsic policy and the SFC policy) at the end of the training (Fig 12 That means the SF are trained with the experiences from the behaviour policy of our agent. We can see that the SD from each coordinate to the starting position tends to grow as the geometric distance increases, especially for those that locate the pathways leading to later rooms. This shows that the learned SD and the geometric distance are in good agreement and that the SF are learned as expected. Furthermore, in the heatmap, which also supports the hypothesis that SFC leads the agent to bottleneck states. We believe this is the first time that SF are shown behave in first-person view environment
observe big intensity changes around the bottlenecks (the
entries)
17

---

Under review as a conference paper at ICLR 2020
as would expect from its definition. The evolution of the SF over time is shown in the video http:::/goo.lee..///2ottp
F APPENDIX: MORE DETAILS ON THE SCHEDULER
When learning optimal value functions or optimal policies via TD or policy gradient with deep function approximators, optimizing with algorithms such as gradient descent means that the policy would only evolve incrementally: is necessary that the TD-arget values do not change drastically over short period of time in order for the gradient updates to be meaningful. The common practice of utilizing target network in off-policy DRL (Mnih et al. 2015) stabilizes the update but in the meanwhile making the policy adapt even more incrementally over each step.
But intrinsically motivated exploration, or exploration in general, might benefit from an opposite treatment of the policy update. This is because the intrinsic reward is non-stationary by nature, as well as the fact that the exploration policy should reflect the optimal strategy corresponding to the current stage of learning, and thus is also non-stationary
With the commonly adopted way of using intrinsic reward as a bonus to the extrinsic reward and train mixture policy top, exploration would be balancing act between the incrementally updated target values for stable learning and the dynamically adapted intrinsic signals for efficient explo- ration. Moreover, neither the extrinsic nor the intrinsic signal is followed for an extended amout of time.
Therefore, we propose to address this issue with a hierarchical approach that by design has slowly changing target values while still allowing drastic behavior changes. The idea is to learn not a single, but multiple policies, with each one optimizing on a different reward function. To be more specific, we assume to have N tasks T E T (e.g. N = 2 and T = (T, Ti} where TE denotes the extrinsic task and Ti the intrinsic task) defined by N reward functions (e.g. Rg and R1) that share the state and action space. The optimal policy for each of these N different MDPs can be learned with arbitrary off- off-policy DRL algorithms. During each episode, a high-level scheduler periodically selects a policy for the agent to follow to gather experiences, and each policy is trained with all experiences collected following those N diffcrent policies. The overall learning objective is to maximize the extrinsic reward (E (9T (S,, A: ~ TT(+(S:)) (w: the macro-policy of the scheduler).
By allowing the agent to follow one motivation at a time, it is possible to have a pool of N different behavior policies without creating unstable targets for off-policy learning. By scheduling M times even during an episode, we implicitly increase the behavior policy space by exponential to M for single episode. Moreover, disentangling the extrinsic and intrinsic policy strictly separates station- ary and non-stationary behaviors, and the different sub-objectives would each be allocated with its own interaction time, such that extrinsic reward maximization and exploration do not distract each other. We investigated several types of high-level schedulers, however, none of them consistently outperforms a random one. We suspect the reason why a random scheduler already performs very well under the SID framework, is that a highly stochastic schedule can be beneficial to make full use of the big behavior policy space.
We investigated three types of high-level schedulers:
Random scheduler: Sample a task from uniform distribution every task steps.
Switching scheduler: Sequentially switches between extrinsic and intrinsic task.
Macro-Q Scheduler: Learn a scheduler that learns with macro actions and from sub- sampled experience tuples. In each actor, keep an additional local buffer that stores N + 1 subsampled experiences: Nm 8t-2m, St-m, 5r). Then at each environ- ment step, Besides the K-step experience tuple mentioned above, we also store an addi- tional maaro-transitie replay buffer. This macro- macro-transition is paired with the current task as its macro-action The Macro-Q Q Scheduler is then learned with an additional output head attached to 0gp (we also tried 0g))
{s:-Nm. $t) along with its sum of discounted rewards to the shared
Threshol--Q Scheduler: Selects task according to the Q-vulue output of the extrinsic task head. For this scheduler no additional learning is needed. It just selects a task based on the current Q-value of the extrinsic head 0.. We tried the following selection strategies:
18

---

Under review as a conference paper at ICLR 2020
Running mean: select intrinsic when the current Q- of the extrinsic head is below its running mean, extrinsic otherwise
Heuristic median: observing that the running mean of the Q-values might not be good statisties for selecting tasks due to the very unevenly distributed Q-values across the map, we choose a fixed value that is around the median of the Q-values (0.007), and choose intrinsic when below, extrinsic otherwise
As report in the paper, none of the above scheduler choices consistently performs better across all environments than a random scheduler. We leave this part to future work.
G APPENDIX: SINGLE RUNS
SP (MUSEI
10
2.
SI0 IMICHI
1.0
H+CC
50x
000
10.0
Figure 13: Learning curves for each run for each agent trained on the flytrap environment.
SID DHLSKI
MUSEC
SE0 INLENN
MIKH
$9.0
MARNOO
Figure 14: Learning curves for each run for each agent trained on the MyWayHome environment.
SI PASKEI
SE0 IMICHI
06
12
15
02
12
060
updens
15
Figure 15: Learning curves for each run for each agent trained on the AppleDistraction envirommeet..........m.mmm
19

---

Under review as a conference paper at ICLR 2020
AND:
M+RUD
101et WoUTt pe S800e
0111
Mi.icar parainttr spdioss
Hurde
Figure 16: Learning curves for each run for each agent trained on the Cartpole environment.
20


Reviews:
[{"review_id": "Syga7o0atr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes the use of a controller that selects whether to act according to a policy trained to maximize an intrinsic reward or a different policy trained to maximize the extrinsic reward of a task. The two policies are trained jointly and off-policy. However, the controller is not trained for their experiments and instead randomly (with equal probability) picks one of the two policies every N steps (with N fixed). They also introduce a new kind of . intrinsic reward based on successor features, that is supposed to capture trajectory statistics for a fixed policy. They name their method scheduled intrinsic drive (SID).\n\nMain Comments:\n\nWhile this paper proposes some interesting ideas,  I am concerned about the soundness of the method, some of the precise implementation details and I believe the empirical evaluation could be greatly improved. \n\nOne of my main concerns is the soundness of using the SFC as intrinsic reward while training off-policy. SFs are defined for a fixed policy so they capture statistics of future states if that policy is being used for control. Can you provide more explanation for why the SFC should still be a useful signal for the agent in the case in which it will follow a very different policy (which seems likely given that the replay buffer not only contains a mix of the exploration and exploitation policies, but also policies at different points during training with potentially very different state visitation distributions). Are the SFs trained with data from both the exploration and the exploitation policy? How can we expect the SFC to have useful signal since it is trained using such a wide range of policies?\n\nIs there any guarantee that the arbitrary feature embeddings (which are not learned in your experiments if I understood correctly) and thus the successor features (SFs) will contain meaningful information about the kinds of states a policy will visit in the future? An ablation using hand-designed feature embeddings that contain relevant information about the state (i.e. in a gridworld) might be useful to understand how it compares to a randomly initialized network, which is what you used for the state embeddings as I understand it.\n\nI am also concerned by the novelty of this work and the fact that it is missing references and discussion to prior work that proposes very similar ideas. For example, [1] proposed the optimization of different losses at the same time: one for exploitation and one or more for exploration. Can you please discuss what is the difference between your method and theirs (other than the intrinsic reward used for the exploration policy)? Similarly, [2] attempts to decouple exploration and exploitation in RL. This reference (and perhaps others that I have missed) should be included and discussed in the paper.\n\nThe empirical validation is missing important statistics such as variance across runs. Experiments on AppleDistractions and Cartpole only have 3 random seeds which I do not think is enough for drawing conclusions confidently. Moreover, on the simpler and standard tasks, SID does not seem to be significantly better than other baselines. It is only on  carefully designed tasks (e.g. FlytrapEscape or AppleDistractions) that are not regularly used as benchmarks that the method seems to perform better. \n\nThe experiments section could be improved by including other (more powerful) baselines such as count/pseudocount exploration methods which have been shown to be more effective than ICM / RND for certain benchmarks, the paper using an intrinsic reward based on successor representations [3] or even Go-Explore [4] that is specifically designed to deal with distractor objects for the AppleDistractions task. Additionally, evaluating SID on harder exploration tasks that are generally considered to be good benchmarks by the community would be helpful (e.g. Montezuma Revenge, Pitfall, sparser versions of DoomMyWayHome etc.) would also strengthen the experimental section.\n\nOther Questions / Comments:\n\n1. There is no measure of the variance / standard deviation across the random seeds in any of the plots. I find it necessary to be included in the plots, along with the mean across runs. \n\n2. What is the reasoning behind using the number of updates (instead of e.g. number of frames / steps / episodes) in the plots? How exactly do you measure the number of updates that appears in the plots? Is that the total number of updates used for the control policy, the exploration policy, and the successor features or is it only the number of updates used for the control policy?\n\n3. I find the use of the term \"hierarchical\" in the title and throughout the paper to be misleading since this term is usually used with a different meaning in the RL literature (i.e. to refer to options/subpolicies that a higher-level policy might choose to pursue at a given time). In your case, the control policy is one of the subpolicies and the other subpolicy is only used for exploration.\n\n4. The paper also contains claims which I find unsubstantiated by the results / analytical formulation such as: \" our proposed SFC reward implicitly captures statistics over the full distribution of policies that have been followed,\nsince the successor features are learned using states sampled from all past experiences\" on page 2 or \"Another valuable property of SFC is that it adapts in very meaningful ways that lead to efficient\nnon-stationary exploration policies, when the transitions gathered by a policy maximizing the SFC\nreward is used to update the SF itself\" on page 5. Please provide more intuition or theoretical / empirical evidence to support such claims. \n\n5. What do you use for the fixed interval (N) at which the meta-controller is choosing which policy to follow? Have you tried training the meta-controller? It would be interesting to see how the results change as N varies. Is N = 1 better than N = length of episode or the other way around or does the choice of N not matter that much?\n\n\n\nReferences:\n[1] Beyer, Lucas, et al. \"MULEX: Disentangling Exploitation from Exploration in Deep RL.\" arXiv preprint arXiv:1907.00868 (2019).\n[2] Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. GEP-PG: Decoupling Exploration and\nExploitation in Deep Reinforcement Learning Algorithms. In Proceedings of the International\nConference on Machine Learning (ICML), 2018.\n[3] Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray\nCampbell. Eigenoption discovery through the deep successor representation. arXiv preprint\narXiv:1710.11089, 2017.\n[4] Ecoffet, Adrien, et al. \"Go-explore: a new approach for hard-exploration problems.\" arXiv preprint arXiv:1901.10995 (2019).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxkKNOh9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: A very nice study on the benefits of successor feature control as an intrinsic drive for hard exploration problems. The work builds nicely on previous work on using SF for exploration and proposes using derived (reachability under $\\psi^{\\pi}$) distances  as intrinsic motivation for an (purely) exploratory policy. This exploratory strategy will be used in conjunction with a policy trained on the extrinsic reward to gather (off-policy) data for both learning processes. The author proposed 'combining' these two policies via a simple scheduler, similar to (Riedmiller 2018).\n\nGood paper/addition to both the intrinsic motivation literature and SFs studies.\n\nPositives:\n1) I like the separation of concerns achieved by training separate policies train for the two reward signals (intrinsic and extrinsic).\n2) SFC as intrinsic reward and the study comparing this with other intrinsic signals (ICM/RND). The more interesting study might in the appendix though (Appendix A). I would suggest moving that into the main paper, as it nice separate the influence of scheduling component and the 'quality' of the proposed intrinsic reward.\n3) Carefully conducted study, with relevant ($\\epsilon$-SOTA) baselines and ablation studies.\n\nPoints of improvement or clarification:\n1) The SFs and the derived reward were done based on random pseudo-rewards $\\phi$ (Pg 5, SF-Nets). It maybe worth exploring learning those to capture more interesting features of the task at hand, especially in situation were there is more signal in the extrinsic reward. Do the authors have a sense of how problematic changing this component throughout training would be? As this acts are a reward signal to the inference the intrinsic reward signal, which then trains the exploratory policy. Thus small changes in one, can have massive implications for the trained Q-net, $Q_{E}$.\n2) It wasn't clear from the exposition which policy is used to train the SFs? The exploratory policy, the uniform random one or the behaviour policy (the combination between the two trained policies given by the Q-nets).\n3) On the SID setup. Did you conduct any studies on M (the number of switches)? For instance, how does this compare with something like episode switching, which has been explored before?\n4) There are a couple of observation/discussion claims that are not really substantiated (for instance, last paragraph in Sec. 3.2). The paper is fine content-wise, without them. I would strongly suggest either removing them, re-phasing them as hypothesis and/or back them by more evidence. \n5) The link to the video (https://gofile.io/?c=HpEwTd.) doesn't work. Please update.\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkeuWsu0qS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Summary\n\nThis paper proposes a novel intrinsic reward for exploration called SFC (successor feature control), to deal with sparse-reward and hard-exploration task. The main idea of SFC is to provide an agent with intrinsic reward defined to be the L2 distance between the successor features of two consecutive states (Equation 4). An underlying motivation of SFC exploration is that high SFC would encourage the agent to enter the \"bottleneck states\" and therefore helps to explore the entire state space. \n\nAnother line of contribution is SID (scheduled intrinsic drive), where a scheduler is used to determine which of the two separate policies (one for extrinsic reward and intrinsic reward) is chosen, with a fixed probability in their implementation, and executed for the next rollout of experience. It has an effect of longer-term exploration and prevents the agent from collapsing to a local-optimum behavior. \n\nEmpirically, the SFC+SID algorithm is evaluated on custom sparse-reward navigation-type environments such as VizDoom and DeepMind Lab (as well as a simple pixel-based continuous control), and outperforms other intrinsically motivated RL algorithms including RND and ICM.\n\n\n\n## Overall Assessment\n\nOverall, I like the idea of this paper and finds it very interesting and promising, but feel it would be on the borderline or slightly below the bar.\n\nThis paper studies a very interesting and novel approach of leveraging successor features for exploration. Successor features are a promising way of learning dynamics-related, task-agnostic representation for RL, which can provide a temporally extended exploration signal. The resulting method presents an improvement over existing intrinsic-reward exploration algorithms.\n\nHowever, I think there are some weaknesses of the paper that would put the paper slightly below the acceptance threshold: empirically the environments are not diverse enough, and they have an implicit structure assumed and favorable for the proposed method --- there remains a question whether the method is general and not task-specific. I also think there are some misleading overclaims. Please see the detailed comment below.\n\n\n\n## Detailed Comments\n\n**[Problem Motivation and Significance]**\nThis paper address an important, long-standing problem in RL of efficient exploration under sparse-reward environments.\n\nA minor comment: in the introduction, it is said that \"terminal reward RL settings\" are considered (the reward is given when the goal is achieved) --- which is an extreme case of sparse-reward RL problems --- but in the experiments non-terminal reward environments are studied, e.g. \"AppleDistractions\" where each of apples yields +0.05 reward. I think the overall claim could be a bit toned down to, for instance, dealing with sparse-reward environments.\n\n**[Clarity]**\nOverall the paper is clearly written and easy-to-follow. Descriptions of implementation details are well provided. However, there are some parts that can be better clarified and improved more. Please see more detailed comments below.\n\n**[Justification of Method (SFC & SID)]** \n\nThe choice of successor feature for driving a novelty-like intrinsic reward signal seems well-motivated. This is because learning of successor features is task-agnostic and only related to multi-step dynamics (though SF is with respect to \"a policy\"), which gives a good representation that captures topological characteristics of the environment. The way intrinsic reward is derived, the squared distance of SFs of $S_t$ and $S_{t+1}$, basically encourages the agent to visit and go across the \"bottleneck\" state. \n\nIn the description of methods, it should be clearly noted that what is the underlying policy being learned for deriving successor features (i.e. $\\pi$ in Eq.3 and 4) --- for example, is it a behavoral policy (which is a mixture of two policies) induced by scheduled drive? Another comment related to this about \"SFC captures statistics over the full distribution of policies that have been followed, ...\" (section 2), which sounds a bit overclaiming to me. Please note that a SF is with respect to a specific policy (e.g. behavioral policy), from the expectation in the definition; I think the use of past experience for minimizing the TD error is basically for estimating the expectation term through approximation, so I am not sure that this claim is well-justified.\n\nIt is not very clear to me why the SFC reward agrees with bottleneck states. I don't think the explanation given in Section 3.2 is logically enough. Also, isn't it only true under a random exploration policy? How would you defined the \"bottleneck states\" (e.g. Tomar et al. 2019) -- which can be helpful for making the main idea more understandable? Moreover, there was no enough explanation or reasoning about why SD (successor distance) is roughly the shortest path between the states.\n\nIn SID, a policy for extrinsic rewards and another policy for intrinsic one are learned. But in case the extrinsic reward was never received (in case of terminal-reward environments), the former policy would behave no different than random policies. Is this interpretation correct?\n\nAlso, given the presented form of SID, it sounds like a bit overclaiming to say it is a hierarchical RL agent, since the scheduler just picks one of the policies with equal probability (rather than being learned) --- especially one policy would become a random exploration policy --- and there is no notion of abstraction or goal/options.\n\n\n\n**[Environment choice]**\nI feel (1) that the environments being evaluated on are not diverse enough, and (2) that the environment in the experiment seems to exhibit specific properties that are favorable to the algorithm.\n\n(Bottleneck State) One implicit assumption is that the structure of navigation is chosen such that following bottleneck states would lead to an optimal trajectory. I agree that even on the maze like FlytrapEscape the navigation/exploration problem is not easy in the absence of rich reward signals, but this is exactly a sort of environments on which SFC can perform better, especially compared to RND/ICM which are not attracted by bottleneck states (Appendix A). It is good though, and could be beneficial in many cases with the presence of bottleneck states, but seems general applicability is a little bit short (not as much as claimed).\n\n(Distinctive appearance) Another assumption is about a choice of appearance. One important thing to note about SF learning is that a feature for state or transition (cumulant) is kept fixed after random initialization, rather than being learned as in (Machado et al. 2018; Kulkarni et al. 2016; Barreto et al. 2017). This is because this method does not need to do regression of reward function. Then, the state-feature $\\phi(s)$ should be discriminative enough so that it can capture some topological and global characteristic of the state space. In general, this is not an easy problem (for first-person view POMDPs), but seems on the environments (FlytrapEscape, AppleDistractions) it was possible because each room/sector has uniquely identifiable wall color and texture. I feel this is somewhat strong assumption made to make SF work. Thus, \"We believe this is the first time that SF are shown to behave in a first-person view environment as one would expect from its definition\" would sound a bit overclaiming. Would this method work on more general environments that do not have this property --- specifically, what will happen if rooms are not distinguishable from color and texture (and the walls were looking similar)?\n\nControl from pixels (DM Cartpole) is an example of environment that does not have these assumptions, but one downside is that action space was simplified and discretized. Indeed, the improvement shown on Cartpole over ICM/RND is not substantial enough. To demonstrate that SFC+SID is \"generally useful\" as claimed in the paper, presenting benchmark results on standard discrete-action Atari environments, or more diverse RL environments would have greatly strengthened the paper to be more convincing.\n\n\n\n**[Analysis of successor distance]**\nFigure 11 (visualization of successor distance) is a great analysis, and I liked it. It clearly shows a smooth topology of the environment thanks to the temporally-extended representation that SF captures. I found that the difference heatmap is a little bit difficult parse. Also, under which policy the SF was computed (I guess this is a behavior policy derived by SID; it should be clearly mentioned somewhere in the paper)? \n\n**[More minor comments about experiments]**\n- Was the same K-step objective (e.g. K=5) used for all of SFC, ICM and RND? If so, what would the result look like when K=1?\n- The ablation study (appendix 1) is interesting and very important. The \"Ours\" algorithm in the main text is actually a combination of SFC and SID, so the comparison shown in this ablation study could be a main result.\n\n\n\n## Feedback for Improvement\n\nMore related work:\n\n* Learning decomposed value functions for extrinsic and intrinsic rewards have been discussed in (Burda et al, 2018b), though in their work a single policy is being learned.\n* [Comparison with Machado et al. 2018] It is discussed that (Machado et al. 2018: count-based exploration with SR) is very similar because of the use of SR/SF. The ways of how to derive intrinsic reward signal are indeed different, but it would be great to have a detailed discussion about how they are different or similar.\n\n\n\nMinor comments:\n\n* Citation needed on section 3.1 --- (Kulkarni et al. 2016 or Barreto et al. 2017)\n* Please consider putting the environment name in the title of each learning curve.\n* Typo: Therfore (right before section 3.2)\n* Typo: temporarily -> temporally (introduction bullet point 2)\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJev5lEbcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of how to integrate intrinsic rewards most effectively, in the episodic sparse reward setting. It has two main technical contributions. The first is Scheduled Intrinsic Drive (SID), which trains two separate policies -- one for maximizing the extrinsic (i.e., task) reward and another for maximizing the intrinsic reward -- rather than a single policy that maximizes a weighted combination of both. This uses the same training setup as prior work, Scheduled Auxiliary Control (SAC), except here the extra policy is trained on intrinsic reward rather than an auxiliary task. The second contribution is Successor Feature Control (SFC), a novel approach for computing intrinsic rewards. For a given transition (s, a, s'), the intrinsic reward from SFC is the squared difference in successor features between states s and s'. Since successor features encompass a notion of which kinds of states the agent will encounter in the future after starting from the current state, this type of intrinsic reward is more far-sighted than most state-of-the-art approaches. Empirical analysis shows that SFC leads agents to explore bottleneck states, which is especially helpful for solving navigation tasks.\n\nThis paper is well-motivated and clearly written. The experimental evaluation of this paper is thorough, comparing SID to adding extrinsic and intrinsic reward together, and comparing SFC to two recent approaches for generating intrinsic rewards, ICM and RND. The appendix does a good job of providing implementation details for reproducibility, in particular regarding reward normalization and the variation of prioritized experience replay. I also greatly appreciate that design decisions are justified, for instance that the choice of using a random scheduler was made because it outperformed several versions of a learned scheduler.\n\nMy only concerns with the paper have to do with evaluation. SFC is compared to prior approaches for computing intrinsic rewards that only take into account transition-level information, whereas SFC takes into account trajectory-level information, and naturally performs better. But there are also recent approaches that do take into account trajectory-level information in different ways, e.g. Savinov et al. (2018). SFC should also be compared to approaches in this category.\n\nI would also like to see an analysis of the failure cases that SFC is vulnerable to. Currently the evaluation domains used, with the exception of cartpole, are all tasks involving first-person navigation. So I wonder whether SFC is most effective (compared to existing approaches) on primarily these tasks in this domain, that are partially observable. It would be nice to see a wider variety of evaluation domains, for instance Montezuma's Revenge, which is frequently used to evaluate algorithms for computing intrinsic rewards, as well as other methods for improving exploration of RL agents. It would be neat if agents trained using SFC are better able to navigate through the doors in this game, since that seems to be a clear example of bottlenecks.\n\nMinor questions / comments:\n- In Figure 1b, why are the values on the four bottlenecks not all exactly the same? The maze is symmetric, so I would expect them to be equal.\n- The plots in Figures 3 through 6 should show the standard deviation.\n\nTypos:\n- Page 2, \"inexplicitely\" --> \"implicitly\"\n- Page 4, \"temporarily\" --> \"temporally\"\n- Page 8, \"carpole\" —> \"cartpole\"", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for intrinsically motivated exploration using successor features by interleaving the exploration task with intrinsic rewards and extrinsic task original external rewards. In addition, the paper proposes "successor feature control" (distance between consecutive successor features) as an intrinsic reward. The proposed method is interesting and it can potentially address the limitation of existing exploration methods based on intrinsic motivation. In experimental results, the method is evaluated on navigation tasks using Vizdoom and DeepMind Lab, as well as continuous control tasks of Cartpole in the DeepMind control suite, with promising results. 

On the negative side, there are some domain-specific properties (e.g., moderate map size with relatively simple structures, different rooms having visually distinct patterns, bottleneck states generally leading to better rewards, etc.) that make the proposed method work well. In addition, off-policy learning of the successor features could be a potential technical issue. Finally, the proposed method is not evaluated against stronger baselines on harder exploration tasks (such as Atari Montezuma's revenge, etc.), thus the addition of such results would make the paper more convincing. In the current form, the paper seems to need more work to be acceptable for ICLR.
Decision: Reject