Deep Learning of Intrinsically Motivated Options in the Arcade Learning Environment
Authors: Louis Bagot
Kevin Mets
Tom De Schepper
Peter Hellinckx
Steven Latre
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
intrinsic motivation
auxiliary task learning
options
atari

Abstract: Although Intrinsic Motivation allows a Reinforcement Learning agent to generate directed behaviors in an environment, even with sparse or noisy rewards, combining intrinsic and extrinsic rewards is non trivial. As an alternative to the widespread method of a weighted sum of rewards, Explore Options let the agent call an intrinsically motivated agent in order to observe and learn from interesting behaviors in the environment. Such options have only been established for simple tabular cases, and are unfit to high dimensional spaces. In this paper, we propose Deep Explore Options, revising Explore Options within the Deep Reinforcement Learning paradigm to tackle complex visual problems. Deep Explore Options can naturally learn from several unrelated intrinsic rewards, ignore harmful intrinsic rewards, learn to balance exploration, but also isolate exploitative or exploratory behaviors. In order to achieve this, we first introduce J-PER, a new transition-selection algorithm based on the interest of multiple agents. Next, we propose to consider intrinsic reward learning as an auxiliary task, with a resulting architecture achieving $50\%$ faster wall-clock speed and building a stronger, shared representation. We test Deep Explore Options on hard and easy exploration games of the Atari Suite, following a benchmarking study to ensure fairness. Our results show that not only can they learn from multiple intrinsic rewards, they are a very strong alternative to a weighted sum of rewards, convincingly beating the baselines in 4 of the 6 tested environments, and with comparable performances in the other 2.

Under review as a conference paper at ICLR 2022
DEEP LEARNING OF INTRINSICALLY MOTIVATED OPTIONS IN THE ARCADE LEARNING ENVIRONMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Although Intrinsic Motivation allows Reinforcement Learning agent to gen- erate directed behaviors in an environment, even with sparse or noisy rewards. combining intrinsic and extrinsic rewards is non trivial. As an alternative to the widespread method of weighted sum of rewards, Explore Options the agent call an intrinsically motivated agent in order to observe and learn from interesting behaviors the environment. Such options have only been established for simple tabular cases, and are unfit to high dimensional spaces. In this paper, we propose Deep Explore Options, revising Explore Options within the Deep Reinforcement Learning paradigm tackle complex visual problems. Deep Explore Options naturally learn from several unrelated intrinsic rewards, ignore harmful intrinsic rewards, learn balance exploration, but also isolate exploitative exploratory behaviors. In order to achieve this, we first introduce J-PER, a new transition- selection algorithm based on the interest of multiple agents. Next, we propose consider intrinsic reward learning as an auxiliary task, with resulting architecture achieving 50% faster wall-clock speed and building stronger, shared represen- tation. We test Deep Explore Options on hard and easy exploration games of the Atari Suite, following a benchmarking study to ensure fairness. Our results show that not only can they learn from multiple intrinsic rewards, they are very strong alternative to weighted sum rewards, convincingly beating the baselines in 4 of the 6 tested environments, and with comparable performances in the other 2.
INTRODUCTION
In Reinforcement Learning (RL), an agent is sequentially given states and needs to perform actions in order to maximize obtained extrinsic rewards Te. The agent is therefore deeply tied to the reward signal, and tends to fail when said signal is sparse or noisy. When the environment is very complex or high-dimensional, it is desirable for the agent to explore in a directed way (Thrun 1992). i.e. ex- plicitly looking for new knowledge and experiences. One of the most common ways to generate such aaskiindependent, directed behaviors is through intrinsic motivation (IM), i.e. an alternative reward signal r; spur curiosity and entice behavior exploration (Oudeyer & Kaplan 2009 Schmidhuberr 2010). IM biologically refers to the natural tendency of organisms to explore.
One of the most common benchmarks for Deep RL agents has been the Arcade Learning Environ- ment (ALE, [ellemare et al. (2013)), consisting of Atari video-games. In order to solve the most challenging, so-called hard-exploration games of the domain (Bellemare al. (2016)) state-of-the- art Deep RL methods have integrated IM in complex learning mechanisms, and finally managed to overcome human-leve level play in all 57 games (NGU and Agent57,, Badia et al. (2020b a)). However, a benchmarking study of IM techniques (benchma rk, Taiga et al. (2019)) shows that standalone IM methods still struggle to perform greatly over all games, i.e. both hard- and easy-exploration games. It is shown that, when same 3 haper-parameter is chosen for all games in the usual expression Tt = r + Bri, there is no improvement over the more traditional methods of exploration.
This wide-spread weighted-sum (MS) scheme = -rí + Br has been discussed to come with sev- eral drawbacks, such as wasting world knowledge and transfer potential, hard fine-tuning of the / hyperparameter. and bad scaling to multiple IM functions. In tabular setting, Explore Options (EO, [Bagot et al.] (2020)) have been proposed as an alternative to the WS to answer these concerns. The agent is divided into the Exploiter, trained exclusively with the extrinsic reward re, and the Ex-

---

Under review as a conference paper at ICLR 2022
plorer, trained exclusively with the intrinsic reward r;. The Exploiter can call the Explorer through an option (Sutton et al. (999). i.c. an additional action, to explore for a fixed amount of time. This decoupling prevents the forgetting of the exploratory behavior, maps naturally to multiple rewards, and theoretically lets the agent explicitly learn when to explore. However, because of their tabular nature, these options are inadequate for function approximation, but it is crucial element al- low the agent to generalize the option over states and effectively learn to balance exploration. In addition, parameter sharing necessary prevent an overhead scaling linearly in the number of agents. We revise Explore Options into Deep Explore Options, new method for combining reward signals in Deep RL. We introduce several key changes to general IM approaches, and showease their performance in the ALE. To provide fair and controlled comparisons, we match the algorithm and hyperparameters used in the benchma rk. The contribution of our work is fourfold:
We revise Explore Options within Deep RL to propose Deep Explore Options as strong alternative weighted sum when using intrinsic rewards, extending the benchmark;
We empirically show several properties of DeepEOs: learning from more than one IM reward at once, selecting which to use or ignore, and extracting the exploiting behavior;
We propose auxiliary task learning of the intrinsically motivated agents, inducing 50%
faster runtime for a similar performance and stronger representation;
We introduce J-PER as an extension of Prioritized Experience Replay (Schaul et al. 2016) for multi-agent buffer sampling.
In Section we provide background build Deep Explore Options. Next, we introduce our method
and discuss the introduced elements in Section 3 In Section 4 we provide experiments in Atari following the benchmark. We go over existing work in the field and discuss desirable properties of methods to combine reward signals in Section 5
2 BACKGROUND AND EXPLORE OPTIONS
2.1 REINFORCEMENT LEARNING, OPTIONS, MOTIVATION
Markov Decision Process (S, A, R. 7) where S is the set of states, A is the set of actions, R C R is the set of rewards, p A, R + [0, is the dynamics function, and y is the discount factor. The goal of RL is to maximize the expected sum of discounted rewards from any starting state.
We use the standard RL setting (uutoon & Barto 2018)), modelling the environment
Options (Sutton et al. (1999) refer to temporally extended actions. An option defined as a triple (I, 3). where I C S is the option's 's initiation set, i.e. states in which the option can initiate; и is the option's policy; and S + [0. 1] is the option's 's termination condition. We assume one several intrinsic reward functions fir (s, a, s') are interested in learning from. These can attempt to motivate directed exploration behaviors, but also any other behavior in the environment. An overview of the literature populating the field can be found in Section 5.2
to generate reward that
2.2 EXPLORE OPTIONS
Explore Options (EO, Bagot al. (2020)) have been introduced sum of rewards, in order to ease fine-tuning and prevent loss of the exploratory behavior. The method consists in decoupling the Agent into Explorer, trained with the intrinsic reward rt, and an Exploiter, trained with extrinsic reward re- Switching from Exploiter to Explorer is done through the Explore Option, which the Exploiter can use any time let the Explorer act for fixed amount of steps Cswiteh- The main intuition is that the Exploiter calls the Explorer to execute directed exploration, while the Explorer teaches the Exploiter how to explore and reach new rewards through off-policy learning. Explore Options have only been introduced in tabular setting, where they can most easily be checked for stability and robustness. However, by design they only make sense in a function approximation setting: the option learning requires generalization in order to call the option in states where little is known, and therefore directed exploration required. In addition, EOs enable learning from multiple rewards, but cannot afford to scale training time linearly in the number of intrinsic reward functions available. Some form of parameter sharing will be required to jump to tens of Explorers.
alternative weighted
2

---

Under review as a conference paper at ICLR 2022
at
Replay Buffer
store
S,R,T
nasign prio
814
env
A
1+1
L
Mini-batch
2-PER sampling
L
learning
Exploiter Tree Explorer Tree
L
Figure 1: Deep Explore Option framework. During interactions, the state is passed through both the Explorer and Exploiter Networks and one of them acts according to the Explore Option. Transitions with intrinsic reward stored in the replay buffer, the corresponding PER priorities assigned by both Exploiter and Explorer according to their respective loss. The mini-batch is then created using transitions according to each agent's priorities following 2-PER sampling of both trees, and used for training of all components.
DEEP EXPLORE OPTIONS AND RAINBOW
Fig. gives an overview of the general Deep Explore Option framework, of which we go into more detail in the following subsections. Implementation details of minor importance reported in Appendix A.I
3.1 ALGORITHMIC BACKBONE & TRAINING
Benchmarking Study & Random Network Distillation Experimental protocols in IM tend to be chaotic, with new approaches usually introduced on arbitrary environments and algorithms, and only compared to a non-IM baseline. TTiga et al. (2019) (benchma rk) provides an experimental setup to fairly compare IM methods the ALE,, using the WS scheme. It uses the Dopamine frame- work (Castro et al. 2018), which provides an implementation of the Rainbow agent (Hessel et al.) 2018) with its majors improvements on DQN (Mnih et al.) 2015); Prioritized Experience Replay (Schaul et al. 2016), n-step returns, and Categorical DQN (C5 (C51,
al. (2117). We match this setting and all hyperparameters using the DeepRL Torch implementation (Zhang 2018). The
benchmark compares several intrinsic reward functions; we focus Random Network Distilla- tion (RND, Burda et al.] (2019b)). as it is the simplest, most used method, and is currently part of fixed, randomly initialized larget net- work f S m and learnt predictor network f S n The method generates an intrinsic reward ri = far (s, a,, 8') = (1) - (5') (=))] by distilling the target into the predictor, using the loss as motivation signal. This will therefore lead the agent towards less-visited states, that still show a high distillation error. In order keep our return estimates within C5I's usual [-10, 10] range, we scale the RND reward by = 0.01. This not to confused with the / from WS, as this is vastly easier to tune: it is only meant to keep the RND returns within reasonable scale and never meant to balance different reward signals.
state-of-the-art methods (Badia et al. 2020b a). RND
Training protocol and Off-Policy Learning We build two separate Rainbow agents, using ar- chitectures and hyperparameters from the benchmark, action to the Exploiter, corresponding to the EO. All observed transitions are added to the same buffer, regardless of the agent currently interacting. We train both agents simultaneously, off- off-policy,
Exploiter and Explorer. We add an
3

---

Under review as a conference paper at ICLR 2022
using the same sampled buffer data, masking the option transitions for the Explorer. The Explorer also is not provided the done signal, since might leak information about the task, rush agents with negative rewards, and interfere with exploration (Burda et al. 2019a).
3.2 J-PER: MULTI-AGENT PRIORITIZED EXPERIENCE REPLAY
Rainbow uses Prioritized Experience Replay (PER, [Schaul et al. (0166) to sample more interesting transitions from the buffer, where the notion of priority of a transition is its associated Rainbow loss. Transition priorities are stored in Sum Tree to enable efficient sampling, Since now have two agents, and therefore two notions of priority, we need to adapt the PER algorithm. One possible method is to duplicate the PER algorithm and pass each agent its own interesting data independently, but this would scale with the number of Explorers in the case of multiple intrinsic reward functions.
Instead, we infer and find that the Explorer and Exploiter might benefit from observing each other's interesting data. Indeed, PER assigns a priority based on the agent's loss, but the transition needs to be observed before the loss is updated so a transition that was of little interest up to a certain point can later become relevant because it stood in rewarding trajectory. Such transitions will not be particularly prioritized by PER. On the other hand, the Explorer is bound to select transitions that are visually appealing, or that lead to visually appealing states. In visual environments like rideo-games, such transitions have strong chance to be relevant regardless of the reward we optimize, as they often correlate with key transitions of the MDP. We execute this idea by keeping two PER priority trees, one for each agent. For a batch size of b, we select a split weight W's E [0, 1] and sample b,w, transitions according to the Exploiter tree, and b, (1 - w.) transitions according to the Explorer tree. This means that both agents will learn off-policy from the same data, and see transitions that the other considers interesting. We refer to this method for 2 agents 2-PER (J-PER in general). Importance Sampling is simply done using the total probability of observing a transition Ty: TP (j) I. (j) + (1 w.) P, (j) where P. (j) and TP; (j) correspond to the probability for Tj to be sampled according to the Explorer's ' and Exploiter's priorities respectively.
3.3 AUXILIARY TASKS
Sharing parameters of the model with auxiliary tasks in RL has been shown to provide a denser learn- ing signal and accelerate learning by building strong environment-awar representation (Jaderberg) ct al. 2017), but the learnt prediction and control models are rarely used beyond enriching the loss.
We propose to consider our Explorer learning, i.e. the learning of fully intrinsically motivated policy or value function, as an auxiliary task. Rather than keeping separate vision modules (Convo- lutional Layers), we share the representation of both agents and only separate the control modules (Fully Connected Layers). The shared visual representation therefore needs to be general enough to be relevant for both our Explorer and Exploiter. Each head and independent parts of the same optimizer can adjust for their own reward scale. In our experiments, this architecture has about 50% faster wall-clock speed than separate networks, as less parameters are used. It is important to note that a goal-oriented architecture such as UVFA (Schaul et al.) (013)) would not allow for efficient transfer learning in our case, because is not trivial to reset the Exploiter behavior while conserving the Explorer behavior. Instead, our multi-headed architecture allows for efficient swapping, isolation or resetting of control heads, meaning that adding an Explorer or switching tasks can be done while preserving both the Explorer behaviors and general visual representation.
We combine the losses through C = Le +AC with À = 1. This, again, is not to be confused with the 3 from WS, since each head adjusts for their reward scales, meaning that À does not carry the duty of reward balancing. In addition, À required no tuning in our experiments, partly because the KL losses of C51 often have similar scales, do not risk one gradient overshadowing another.
3.4 MULTIPLE EXPLORERS
Learning from several intrinsic rewards Intrinsic reward functions fir are often based on heuris- ties and intuition regarding what signal would motivate the agent to produce the most interesting and beneficial behaviors. The range of possible behaviors produced by Intrinsic Motivation is extremely wide and widens with the complexity of the environment in a Visual Navigation setting for exam- ple, we can motivate going fast, grabbing or observing objects, covering ground, changing rooms,
4

---

Under review as a conference paper at ICLR 2022
opening doors... Therefore, it becomes relevant to learn from multiple (say, J) intrinsic rewards at once, since even exploration-drive rewards might complement each other in different settings and environments (Matusch et al., 2020). The WS approach very hardly scales with several intrinsic reward functions Sir.j. since it would lead to the weighed sum R = Re Σ3 83 Rs.j. where the tuning of all 3, and the cacophony of all reward signals would make finding a stable agent virtually impossible. We can simply visualize that an agent with sseeed + T grrah objects would make for very distraught joint behavior. At the other end, EOS scale naturally with several fir.j. by simply adding options and associated Explorers per j. The only practical issue will arise in edge cases where J dominates the action space, hence primitive action learning might be impeded, or if the scarce rela- tive interacting of all Agents begets the necessity of Offline RL (Fujimoto et al.. [2019] Levine al., 2020
ConstNeg intrinsic reward Because the amount of possibly interesting intrinsic rewards is so vast, some are bound to be sub-optimal or even harmful to our agent, which will therefore need to be able to ignore them and focus on the most relevant signals. To demonstrate this, we introduce ConstNeg constant negative intrinsic reward for = -0.025 that behaves like living reward. Crucially, we do not provide the done signal to indicate life loss. For agent that cannot dif- ferentiate between termination signals between the reward functions, like a naive WS, ConstNeg would simply hurt performance by rushing the agent towards life loss Instead, without the termi- nation signal, the agent will be pushed off the starting point and have to explore to make sure that all regions of the state space are harmful. While this can originally help exploration, the agent will quickly learn that no behavior leads to reward, eventually rendering the introduced behavior useless. In other words, this naive and vovial-to-implmmen function induces a behavior that we might want to learn from, but not heavily rely upon. With it, we mean to show that (i) a single intrinsic reward function does not need generate all interesting behaviors, so long as we can combine it with other intrinsic reward signals; therefore (ii) method to combine reward signals needs to be able to scale to several signals, and be robust to harmful signals. Our experiments with this intrinsic reward will show that Deep Explore Options allows for both.
EXPERIMENTS
4.1 EXPERIMENTAL. PROTOCOL
We evaluate Deep Explore Options with RND with the same core algorithm, architecture and hy- perparameters as the Rainbow implementation from the benchmark. Architectural details can be found in Appendix A.22 We compare our method to two baselines provided in the benchmark: Rainbow with c-greedy ("rainbow") and Rainbow with an RND intrinsic reward using ws ("rainbowtrnd") We refer to our method as "DeepEO". Agents are trained over 5 seeds for up to 100 million frames on 6 of the most relevant games of the benchma rk- games in which methods performed most differently, indicating difficulty and importance of the IM approach. We use 3 hard-exploration games: MONTEZUMA'S REVENGE, GR PRIVATRÉYE; and 3 easy- exploration games: SPACE INVADERS, ASTERIX and SEAQUEST. Following the benchmark k and recommended practices (Machado et al., 2018), we use 0.25 aticky-actions and no termination on life loss. The hyperparameters we introduced take values Cswitch = 15 and w. = 0.1 unless stated otherwise, and are studied in subsection 4.3
4.2 RESULTS- - ARCHITECTURES
We start by comparing our main DeepEO architectures. First, the Separate Exploiter and (RND-based) Explorer networks; then the Shared Exploiter-xxplore visual representation (sec- tion 3.3): and finally the Multiple Explorers, i.e. branching to our Shared representation an Explorer trained with the potentially harmful ConstNeg intrinsic reward (section 3.4). We carry this study out on GRAVITAR and MONTEZUMA, as they provide the most disparity in performances of the baselines. We then perform additional experiments on the 4 other games with Mult t only, as our main contribution. We compare DeepEo Muiti i to the two basclines from the benchmark, ra rainbow and rairrabooiai rainbowtrnd.
In AVITAR and MONTEZUMA, the optimal weighted-sum policy with ConstiNe is to die quickly.
5

---

Under review as a conference paper at ICLR 2022
Spacelnvaders
Seaguest
Asterix
1000
12000
00
04
18
PrivatoEye
Montezumaa's 's Revenge
Gravitar
18200
2000
Manbar meme
Nor rete
Natber
Gravitar: w
GraviLar: c switch
rainbow
rainbow
DeepEO Separate
DeepEO Shared
DeepEO Multi
200
1:3
Desibo Laild-
Exploiter only
0.
Menber
Figure 2: Deep Explore Option architectures against Rainbow baselines in Atari. Shaded areas rep- resent standard deviation over runs (3 for hyperparameter studies). Top row: easy-exploration games; middle row: hard-exploration games. Bottom row, left: hyperparameter studies for w. and Czwitch in Gravitar.
Architectures Comparison As can be observed in Fig 2 all DeepEO architectures outperform both baselines in GRAVITAR, while achieving good performances in MONTEZUMA, with the better overall performances.
The Shared and Multi i appear slightly less performant but very comparable to Separate, How- ever again, this architecture is 50% faster for 2 agents, about 100% faster for 3 Agents, and builds a stronger representation we believe the choice should be obvious, especially as we learn from more intrinsic reward functions. Crucially, Multi was able to learn consistently even with the ad- dition of the naive ConstNeg both to its behavior (as an Explorer) and visual representation (in the shared architecture). We believe this is major step from the Weighted Sum, which wouldn's 't be able to adjust to this reward. It is interesting to notice that Multi even seems slightly more robust than Shared; if this additional Explorer could act some form of regularization. This will be investigated in future work.
Baslines Comparison We now widen our focus to the 6-game results of Multi versus ra rainbow and rainbowtrnd. We additionally report Exploiter-only performance, i.e. isolating the Exploiter to only access primitive actions, in order to extract the learnt reward-maximizing behavior. This is obtained an evaluation fashion, every million steps over 5 episodes, with the current 6.
We observe that the Deep Explore Option tremendously improves on the weighted sum, with very clear domination on most games. The method is also consistently faster in early training, reliably finding sources of reward faster than the baselines. Very interestingly, the Exploiter-only testing performance vastly improves the results on casy-exploration games, but is much worse than the training performance in hard-exploration games; with GRAVITAR intuition, as earl-axploottion games are the ones where we most expect the options to be important, hence stripping the agent off of its Explorers there should have the most impact.
outlier. This aligns with
In order to provide more intuition into the Explore Option and Explorers' workings, we provide
Q-value estimates and interaction frequencies in Appendix B

---

Under review as a conference paper at ICLR 2022
4.3 RESULTS- HYPERPARAMETERS
W, hyperparameter Wy controls the 2-PER ratio of transitions interesting to the Exploiter or Explorers that passed for off-policy learning- wi means passing only transitions selected by the Exploiter, i.e. for the task reward, while wi = 0 would mean passing transitions only relevant for the Explorer- intuitively, where new data can be observed. This hyperparameter turns out to be of importance in the performance, but required little tuning. We tested it over 3 seeds and the values w, E {0.1, 0.5, 1} on Gravitar and report the impact on performance on the Separate architecture. Note that we do not study w, = 0 since this would prevent the Exploiter from learning the option, as the Explorer ignores it.
Hyperparameter studies are performed over 3 seeds on GRAVITAR and can be seen in the bottom row of Fig 2 left for wa. We observe that the agent does not benefit from observing a lot of exploiting data (w, RJ 1), but rather benefits from observing data selected by the Explorer. We infer that this because Explorers are generally attracted to visually appealing transitions, which are often very relevant in video-gat -amee,, while the Exploiter's priority for this transition yet has to be updated to reveal interest.
Csuritch hyperparameter In Bagot et al. 2020), the authors linked EO's Couitth hyperparameter to WS' B, since both dictate the intensity of exploration in their respective approach. It was shown that Cowritch was much more robust and lead to overall better performances than any value of 3 in simple setting. While we do not have access the benchmark" 's per-3 performance, we replicate this study in Atari by comparing two different values of Cawitch- It is expressed in number of steps the Explorer acts, so we study Cswitch E {15, 50}.
Results can be seen on the bottom row of Fig 2 middle. We observe that a higher value of Cawstch seems severely harm performance; we infer that this is due to higher option-value variance that makes learning the option less stable. Indeed, unlike tabular setting, in a complex visual setting and with function approximation, the option harder to predict and the agent will attempt to generalize. We advocate for smaller values, they are casier to learn; in addition, they can cover all behaviors generated by the higher values, if the Exploiter learns to spam the option when is relevant.
5 RELATED WORK & DISCUSSION
5.1 EXPLORATION WITHOUT INTRINSIC MOTIVATION
Methods that do not incorporate Intrinsic Motivation were state-of-the-art on Atari until recently. Such methods generally injected noise the action or parameter space to explore. In Value-based methods, this concerns mainly e-greedy (Mnih et al.] 2015), which randomly explores with some probability at each step; and Noisy Nets (Fortunato et 2018 Hessel et al. [2018). which introduce learnt noise in the parameters. In Policy-based methods, entropy maximization (Haarnoja et al. 2018) takes advantage of the stochastic policy formulation to motivate noisier policies.
5.2 INTRINSIC MOTIVATION
IM (Aubret et al.) 2019) is generally used for knowledge acquisition about the environment, but can also encompass skill learning (Eysenbach et al. 2019). where the agent learns a set of temporally- extended actions to help solve the task. In knowledge acquisition, motivating exploration is by far the most common role of IM, and is often divided into state novelty which encompasses count- based methods (Bellemare et al.) 2016) OStrrvvki et al.) 2017: Burda et al.) 2019b) prediction error (Pathak et al. 2017) and information gain (Schmidhuber, 2010). A notable recent method is Never Give Up (NGU, Badia et al. (2020b)), combining intra-episodic and inter-episodic intrinsic rewards to motivate thorough exploration. The 3 fine-tuning is done by a spectrum of actors, taking advantage of the UVFA goal-oriented framework to pass 3 as input (Schaul et al., 2015)). Agent57 (Badia et al. 2220a) improves on NGU by focusing the fine-tuning on the most relevant values using a Multi-Armed Bandit setting, and separating the value functions to provide stability.

---

Under review as a conference paper at ICLR 2022
Method Name
Value
IM usage
Exploratory behavior
Transfer
Multiple
Auxiliary
Functions
IM signals
Task
benchmark
Combined
WS
Merged
No
No
No
RND
Separate
WS
Merged
No
Yes
Explore&Exploit
Separate
Decoupled
Random switch
Yes
Yes
No
NGU
Combined
WS
Merged
No
No
Yes
Agent57
Separate
Merged
Yes
No
No
DeepEO
Separate
Decoupled
Option call
Yes
Yes
Yes
Table 1: Comparison of methods to combine intrinsic and extrinsic rewards
5.3 AUXILIARY TASKS AND PARAMETER SHARING
As mentioned in Section 3.3 auxiliary tasks can accelerate learning, build stronger representation, and therefore improve generalization. UNREAL (Jaderberg et al. 2017). the task is learnt along with several prediction and control tasks such as reward prediction, pixel control and feature con- trol. In Lample & Chaploi (2017), the architecture learns to predict several features of the environ- ment along with the true task and this dramatically improves performance. Note that the auxiliary tasks are only meant to enrich the loss and rarely used to actually predict or control relevant ele- ments. In different line of work, the UVFA framework (Schaul et al. 2015) introduces the agent's goal as input and effectively shares all network parameters between the different goal-conditioned value functions, which has been used in NGU range from exploiting very exploratory policies. Agent57 argued against it, raising the point that decoupling would stabilize learning, allow the func- tion approximators to accommodate their respective reward scales and decouple the optimizer state. Our Shared architecture allows to benefit from the auxiliary task of intrinsic reward learning while addressing the problems raised by Agent57, since each head and associated optimizer can adjust for its own reward scale.
5.4 OPTIONS FOR EXPLORATION
Using options for exploration has been probed in prior work, and is often addressed within option- or skill-discovery, with strong ties to skill learning. The main body of work involves reaching states with key properties. Machado et al. (011)) show that Proto-Value Functions (Mahadevan) 2005) can highlight key states using the MDP Laplacian, and use intrinsic rewards ("eigenpurposes" to generate eigenoptions to reach said states. Covering Options (Jinnai et al., 2019a b) use algebraic connectivity to identify the best states to link with options in order to minimize cover time, and build said options with a goal-oriented intrinsic reward. Successor Options (Ramesh al. 2019) identify landmark states that maximize the successor representation, and reach said landmarks using hill- climbing intrinsic reward. Because they wind down to intrinsic reward learning, all of these methods could be incorporated as Explorers in DeepEOs. From another angle, Nachhum et al. (2019) attempt to understand successes of Hierarchical RL, and identify temporally-extender exploration as po- tential key element, further motivating work. Supporting that claim, Dabney et al. (2021) show that simply repeating actions can be a powerful motor for exploration this could be incorporated in deterministic Explorer.
5.5 COMBINING INTRINSIC AND EXTRINSIC REWARDS
Combining different sources of reward problem that has recently gained attention with the rise of more potent IM methods. In Table I we analyse several methods previously introduced, using key desirable properties to provide systematic comparison.
The first column indicates whether the intrinsic and extrinsic Value Functions are separated or not, ie. using r = Te + 3r; or separate V. and V,, which can be combined into V, + 3V, in WS setting. As far as combine different discount factors and termination signals. Agent57 later proved that separating the Value Functions stabilizes the training, by allowing the function approximators and optimizer states to accommodate their respective reward scales. This issue was conjointly raised in the original Explore Options paper, proposing to decouple the policy-level.
aware, this idea was first introduced for IM in RND (see 3.1 to
8

---

Under review as a conference paper at ICLR 2022
The second and third columns show how each method uses the IM signal explore. WS methods perform a weighted sum of the extrinsic and intrinsic signals. which results in a single merged exploring-exploitie behavior. Explore&Exploit (Nachum al.) 2019) was introduced in vastly different context, to show that Hierarchical RI. mainly benefits from exploration. The algorithm also trains intrinsically motivated Explorer and extrinsically motivated Exploiter and randomly switches the actor every Cowstch steps. DeepEOs are similar but switch agents using a learnt option.
Column 4 indicates whether the method could transfer the learnt exploratory behavior to a new task. All methods with Combined value functions fail this, since the exploring behavior is engraved into the policy. Since RND separates the value functions at the Critic level, it is unclear whether it could benefit from the learnt V,, Other methods that keep separate value functions can simply extract the learnt Explorer to the new task.
Column 5 indicates whether the method could scale to multiple IM functions fir- WS methods fail here, since would require merging more reward signals and additional layers of 3 fine-tuning. Explore&Exploi and DeepEOs naturally scale to multiple IM signals, as it respectively involves switching between more agents, and adding more options.
The last column indicates whether the method shares parameters between the intrinsic and extrinsic reward learners, resulting in an auxiliary task that can improve the representation. We argue that the naive WS does not share parameters since it is learning a single value function. Similar to our approach, RND separates the approximator heads, resulting in a shared visual representation. NGU shares all parameters using UVFA. Both Explore&Exploit and Agent57 could benefit from the shared architecture we introduce in DeepEO.
5.6 LEARNING FROM MULTIPLE SIGNALS
In Horde (Sutton et 2011 the authors propose to build the agent's world knowledge by learning a wide variety of prediction and control tasks that can be useful to learn more complex downstream tasks. This knowledge is represented under the form of value functions, and each learner is referred to as either a control or prediction demon. Deep Explore Options are closely linked to the Horde architecture, due to their scalability to several intrinsic reward functions and independent modules. be seen control demons alongside the Exploiter. Intrinsic reward
In particular, Explorers functions für often predict or model elements containing information about the environment, such as the successor representation (Machado et al. 2020 an action-prediction and forward model (Pathak et (017), or density models (Bellemare et al. 2016). All of these can therefore be seen as prediction demons the Horde architecture. In addition. all of these could be trained using huge shared representation as auxiliary tasks, and hence building both a world and state representation the same time. Under this scope, the resulting agent would make direct use of all its demons, providing one solution as to how to use world knowledge - this will be the subject of future work.
6 CONCLUSION
Explore Options allow an extrinsically motivated Exploiter agent to call an intrinsically motivated Explorer agent for a set amount of time. They have only been conceptualized in tabular setting, preventing generalization of the option. We revised them to Deep Explore Options in the Dopamine framework, to learn from intrinsic reward in Deep Reinforcement Learning. We introduced new multi-agent buffer-selection algorithm, showing that agents can benefit from observing data that is interesting accoding to others. We also introduced a new architecture to combine intrinsic and extrinsic agents, resulting in 50% faster runtime and stronger representation. To ensure fair com- parisons, we followed a benchmarking study, using hard- and easy-expe exploration games of the Arcade Learning Environment. We showed here that Deep Explore Options are very strong alternative to the wide-spread practice of a weighted sum of rewards, convincingly beating the baselines in 4 of the 6 tested games, with a good performance on the last two. We also showed that Deep Explore Options can efficiently learn from several intrinsic reward signals, ignore harmful signals, and ex- tract their exploiting behavior; all while maintaining a strong shared representation. We compared our introduced method to existing work in intrinsic motivation and discussed desirable properties of methods combine extrinsic and intrinsic signals.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Arthur Aubret, Lactitia Matignon, and Salima Hassas, A survey on intrinsic motivation in reinforce- ment learning. arXiv preprint arXiv: 069706676, 2019.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR, 2020a.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020b.
Louis Bagot, Kevin Mets, and Steven Latré. Learning intrinsically motivated options to stimulate policy exploration. Fourth Lifelong Machine Learning Workshop at International Conference on Machine Learning, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Conference on Neural Information Processing Systems, pp. 1471-1479, 2016.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International Conference on Machine Learning, pp. 449-458, PMLR, 2017.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In International Conference on Learning Repre- sentations, 2019a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference Learning Representations, 2019b.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018.
Will Dabney, Georg Ostrovski, and Andre Barreto. Temporally-extended s-greedy exploration. In International Conference on Learning Representations, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representa- fions, 2019.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learning Representations, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062. PMLR, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. 2017.
10

---

Under review as a conference paper at ICLR 2022
Yuu Jinnai, Jee Won Park, David Abel, and George Konidaris. Discovering options for exploration by minimizing cover time. In International Conference on Machine Learning, pp. 3130-3139. PMLR. 2019a.
Yuu Jinnai, Jee Won Park, Marlos C Machado, and George Konidaris. Exploration in reinforcement learning with deep covering options. In International Conference on Learning Representations, 2019b.
Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learn- ing. In AAAI Conference on Artificial Intelligence, volume 31, 2017.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprini arXiv:2005.01643 2020.
Marlos C Machado, Mare Bellemare, and Michael Bowling. A laplacian framework for op- tion discovery in reinforcement learning. In International Conference on Machine Learning, pp. 2295-23004. PMLR, 2017.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Marlos C Machado, Marc Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In AAAI Conference on Artificial Intelligence, volume 34, pp. 5125- 5133, 2020.
Sridhar Mahadevan. Proto-value functions: Developmental reinforcement learning. In Proceedings of the 22nd international conference on Machine learning, pp. 2005.
Brendon Matusch, Jimmy Ba, and Danijar Hafner. preprint arXiv:2012.11538 2020.
Evaluating agents without rewards. arXiv
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Andrei Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 51895540 2015.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? CoRR, abs/1909.1066 2019.
Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In International Conference Machine Learning, pp- 2721-2730, 2017.
Pierre-Y Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Rahul Ramesh, Manan Tomar, and Balaraman Ravindran. Successor options: An option discovery framework for reinforcement learning. arXiv preprint 1rXiv: 2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International Conference on Machine Learning, pp. 1312-1320, 2015.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations, 2016.
Jürgen Schmidhuber. Formal theory creativity, fun, and intrinsic motivation (1990-2010). JEEE Transactions Autonomous Mental Development, 2(3):200-247, 2010.

---

Under review as a conference paper at ICLR 2022
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Richard S. Sutton, Doina Precup, and Satinder P. Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artif. Intell., 112(1-2:181-211, 1999.
Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from un- supervised sensorimotor interaction. In International Conference on Autonomous Agents and Multiagent Systems, volume 2, pp. 761-768, 2011.
Adrien Ali Taiga, William Fedus, Marlos C Machado, Aaron Courville, and Mare G Bellemare. On bonus based exploration methods in the arcade learning environment. In International Conference Learning Representations, 2019.
Sebastian Thrun. Efficient exploration in reinforcement learning. Technical Report CMU-CS-92- 102, Carnegie Mellon University, Pittsburgh, PA, January 1992.
Shangtong Zhang. Modularized implementation of deep rl algorithms in pytorch. https:// github. comm//Siaangtonnggheen///eeeg//p 2018.
12

---

Under review as a conference paper at ICLR 2022
A
IMPLEMENTATION DETAILS
A.1 MINOR ALGORITHMIC CHANGES
A.1.1 FIXING E-GREEDY ACTION-SELECTION
<-greedy is one of the most naive exploration methods, where we simply uniformly try out a random primitive action with probability e. Since our option persists over Csunttch steps, while other actions only last 1 step, this needs to be balanced for. We provide a more complete and precise understanding of the Pew hyperparameter provided the original Explore Options paper, which allows for a more logical choice of its value.
What is desirable not uniform action selection, but uniform agent interaction when explor- ing with e-greedy, since towards the earlier stages of learning, we do not know which agent will best find rewards. We can achieve this simply by tweaking the option probability Per (de- faulting at 1/ (IAI + 1)). When N steps are performed randomly by the Exploiter,
expected
NennCswiteh are performed by the Explorer. To achieve a ratio of r = 50%, we therefore need r / (1 + PezCewiithh): replacing Pea = P/Csiiteh gives p r/ (1 r). We use linear schedule of from 0.5 to 0.001 over 200 million steps to ensure that the call to the Explorer is frequent at first, but eventually up to the Exploiter.
A.1.2 n-STEP AND OPTION LEARNING
Rainbow uses nestep learning, which bootstraps Q value estimates after n steps of real-world tran- sition data. This is well-known achieve a balance between bias and variance for the right value of n. In our case, this means that we need to learn from sequences of n transitions in which the option can appear among primitive actions (e.g. - pirically find that the agent does not benefit from such n-step learning of the option. We infer this because adds dramatic variance to all primitive actions, as the option's value is hard predict. Therefore, the n transitions before bootstrapping do not contain the option and the option is only learnt using one-step transitions. So in practice, when the Exploiter uses the Explore Option, the Ex- plorer acts for Cawitch time-steps, then gives the control back to the Exploiter. This aption-transition (s, a, r, s¹) (st, 0, R, St+cowitthh) with R = Σ k==
a
a
3). We em-
-1 Re+x+1 1 is added to the buffer like other transitions, simply adjusting the discount during bootstrapping.
A.2 HYPERPARAMETERS & ARCHITECTURES
Hyperparameter
Value
Discount factor 7
0.99
Min history to start learning
80K frames 32K frames
Target network update period
Adam learning rate
6.25 5
Adam e
1.5 1.54
Multi-step returns n
3
Distributional atoms
51
Distributional min/max values
[- 10, 10]
e-greedy schedule
1 0.01 over 1M frames
Table 2: List of Rainbow hyperparameters used, taken from the benchmark
The list of used Rainbow hyperparameters can be seen in Table above. The network architecture used for the Separate configuration is the Categorical DQN, as used in Dopamine (Rainbow without Dueling). The network has 3 Convolutional layers followed by 1 fully-connected layer, which we refer to as the "vision module", followed by a fully-comnected output layer, which we call "control module". The Exploiter has one more action per Explorer. Instead of keeping separate network for Exploiter and Explorers, the Shared and Multi i architectures simply share the vision module between all agents and branch out control module for each agent.
13

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL RESULTS & VISUALS
B.1 EXPLORER & OPTION INSIGHTS
The Explorers are also implemented as Rainbow agents using their respective intrinsic rewards as learning signals, and without a done life-termination signal. When their respective options are cho- sen by the Exploiter, they are called to act for Cywitch steps. We can therefore track how frequently they act in the environment, providing insight as to how relevant each Explorer is to the Exploiter. We can also keep track of the Explorer's Q-values during training (on training batches), giving an idea of how interested each agent is in the environment at each step of training. We compute both metries for each Explorer every million steps.
The visuals for the Multi architecture and for all games provided in Figure 3 with Explorerl (left column each game) referring to the RND-based Explorer, and Explore12 (right) referring to the Cons t setee-bbaedd Explorer.
There are several key observations to be made first, we can infer how interesting game was to the Explorers based on their Q-value estimates (top row of each game). The ConstNee-bassee Explorer in particular allows us a degree of interpretability: if the Q-value estimate skims the 225 mark, means the Explorer is not interested all in the environment and manages correctly predict its reward function. However, in certain games, especially hard-exploration ones (PRIVATEEYE, MONTEZUMA, GRAVITAR), we see tendency to overestimate the value of the observed states. We understand that this is because these games display new visuals on rare occasions, which makes these parts of the input space hard predict. Interestingly, the RND-based Explorer seems to align with the CoonstNeg-baace Explorer, as it has more noisy and generally higher Q-value estimate the same games.
To certain extent, we can also infer how relevant the Exploiter finds each Explorer based on the interaction frequencies, i.e. how many times the Explorer acted the environment over period (bottom rows). These vary wildly, between near-zero in some games to up 20% in others. This, however, more likely provides an idea of how indulgent the environment is, as very hard environ- ments wouldn't allow for non-exploiting policy to be tolerable (leading death or huge perfor- mance losses). So this measures how much the agent can afford to explore without big losses of performance. A better measure of the relevance of the Explorers the Exploiter the direct ratio of performance of the Exploiter with and without Explorers, as reported in the main text in Figure Combined with the observations of Figure here,, also hard to navigate, since they allow for little leeway for Explorer interactions, but they absolutely require the Explorers, since the Exploiter performance crumbles without them. In casy-exploration games or GRAVITAR (which is more arguably hard-exploration) the Exploiter can afford to use the Explorers much more, but doesn't 't require them as critically, in fact performing much better in iso- lation. Still, Explorer interactions of more than 5% show that the Exploiter finds clear utility to using the Explorers.
that hard-exploration games tend be
14

---

Under review as a conference paper at ICLR 2022
Spaceliivdders
Seaquest
PrivateBye
Montozuma's Revenge
Gravitar
Figure 3: Explorer interaction frequency and Q-value estimates in all games for the Multi archi- tecture.
15


Reviews:
[{"review_id": "wRGDII5UDN", "reviewer": "Reviewer_zfem", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Important problem, definitely of interest to the ICLR community\n- Care seems to be taken to follow best practices with regards to conducting RL experiments (random seeds & using standard implementations)\n\nWeaknesses:\n- Limited performance improvement in the settings where the greatest benefit would be expected (hard exploration problems)\n- Limited technical novelty\n- Experiments could be more comprehensive (only compared with 2 baselines, ablations are relatively limited)\n- Some claims (e.g., \"improved representation\" and instability of weighting multiple IM objectives) are not clearly backed up by experiments\n\nOther comments:\n- \"We combine the losses through L = Le + λLi with λ = 1.\" The loss terms here are not defined\n- \"This, again, is not to be confused with the β from WS, since each head adjusts for their own reward scales, meaning that λ does not carry the duty of reward balancing.\" This is still a sort of balancing, because if the rewards for explorer are much larger/smaller than the exploiter, the gradients contributions will also be much larger/smaller, right?\n- \"All observed transitions are added to the same buffer, regardless of the agent currently interacting.\" Which reward is stored in the buffer? Or are both rewards stored?\n- \"where the tuning of all βj and the cacophony of all reward signals would make finding a stable agent virtually impossible\" It would be nice to back up this claim with experimental evidence\n- It's hard to interpret the results of Figure 2- on one hand, max(Exploiter only, DeepEO Multi) seems to be fairly strong; however, I'm not sure this is the best interpretation. It seems problematic that in some cases, the exploitation policy is much stronger than the explore option policy, while in others, relying on the exploration policy is necessary to achieve good performance. In practice, which should be used when the policy is deployed?\n- The algorithm introduces 3 additional hyperparameters, \\lambda, c_{switch}, and w_s. They don't seem to require significant tuning, but given that the algorithm is partially motivated by the difficulties incurred by the weighting parameter in standard intrinsic motivation methods, it seems worth mentioning as a minor downside.\n- \"Our Shared architecture allows to benefit from the auxiliary task of intrinsic reward learning while addressing the problems raised by Agent57, since each head and associated optimizer can adjust for its own reward scale.\" I'm unconvinced this is true- even with separate heads, since the majority of the network is shared, aren't different reward scales a problem?\n- \"...resulting in an auxiliary task that can improve the representation.\" This claim doesn't really seem to be clearly supported by the experiments, unless I'm missing something\n- The primary benefits of \"resulting in a 50% faster runtime and stronger representation\" don't seem to be systematically evaluated in the experiments.\n- \"...convincingly beating the baselines in 4 of the 6 tested games, with a good performance on the last two.\" It's counterintuitive that the main benefit from DeepEO came on the *easy* exploration tasks, rather than the hard ones. Can the authors explain why this occurs? It seems counter to the idea that separating IM is important for exploration.", "summary_of_the_paper": "The paper is motivated by the challenging problem of exploration in reinforcement learning. The authors build on previous work in learning exploration options, extending it to the setting of function approximation and learning from image-based observations. The primary contributions of the work are the function-approximation version of Explore Options, called Deep Explore Options (DeepEO), which learns a separate exploration policy alongside a pure exploitation policy. In addition, the authors describe the training strategies/data sampling mechanism needed to train DeepEO successfully.", "summary_of_the_review": "Overall, the authors attack an important problem of interest to the ICLR community. While the idea of using options for better exploration in deep Q networks is interesting, the proposed method does not currently provide clear performance improvements over existing approaches in the experimental settings shown, and the experimental settings themselves were relatively limited in terms of the diversity of baselines considered. In addition, some claims made in the paper are not fully backed by experimental evidence. Thus, I recommend the current version of the paper be rejected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RdF_J0UT2QL", "reviewer": "Reviewer_yuhD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. \n\nThe explore options (EO) or deep EO generalization done in this paper seems closely related to semi-MDPs. There is no reference or contextualization w.r.t this literature. \n\nWhy is a new formalism needed if it can already be formalized as an semi-MDP? If semi-MDPs are not rich enough to describe these ideas then what is missing? \n\n2. \n\nFigure 2 provides quantitative results and I was surprised to see that the scores on montezuma's revenge are not as high. \n\nThere have been other approaches (e.g. Go-explore) that has surpassed this by a wide margin or other hierarchical RL approaches (e.g. h-DQN) that have studied ways to explore more sample efficiently. \n\nOn a related note, I think the biggest limitation of this paper is that the experimental validation is not thorough. Where were these particular 6 games selected? It is difficult to quantify the merits of this approach without running this on wider environments. \n\n3. \n\nSince the key idea is to learn options, it would have been interesting and important to visually inspect the type of intrinsic options that are learned by this approach. The authors provide a partial answer via Figure 3 in the appendix but this does not clearly indicate the exploration abstract learned by this approach\n\n4. \n\nIs the exploration option's termination condition always deterministic? This seems like a big limitation in both discrete and continuous domains. What are the effects of changing this parameter on the performance beyond Gravitar? ", "summary_of_the_paper": "This paper proposes a directed exploration algorithm within the context of Deep RL. It learns two separate off-policy agents -- one learning to explore and the other one to exploit given task-specific rewards. The explorer is task-agnostic and assumes that a set of intrinsic rewards are provided. The exploiter can execute raw actions or options parametrized by the explorer. The approach is shown on a small set of atari games, where the key performance metric is the highest score reached within 1e8 number of frames. \n\n", "summary_of_the_review": "I think the ideas presented in this paper are interesting and promising. However, I would like to see them validated on more than 6 ALE games -- currently the results are somewhat mixed (helps on 4/6 to some extend) and there isn't an empirical result that very clearly demonstrates a new capability enabled by this approach. \n\nThe paper already needs to be more clearer written and framed w.r.t a rich literature on semi-MDPs and hierarchical RL. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QnHAG_-SIf", "reviewer": "Reviewer_jgim", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduces a new agent architecture that has several moving pieces and it doesn’t do a good job at evaluating their impact. In the end it is hard to claim that the value of the main idea (to use an intrinsic reward signal without having to linearly combine it to the extrinsic reward) was backed up with clear experiments. There are several confounders. For example, it could still be that the observed empirical performance is due to some sort of temporally-extended exploration, and that’s all. I don’t see data to rule this out. Finally, the paper makes several claims that are not precise or that are not supported by data, including some of its claimed contributions. I elaborate on these topics below.\n\nFirst and foremost, I don’t think we have data to quantify the impact of several components discussed in the paper. Dabney et al. (2021) has shown the benefits of repeating the same action multiple times to generate temporally-extended exploration. When I visit their paper, and I look at their learning curves (Fig. 19 from this version https://openreview.net/pdf?id=ONBPHFZ7zG4), I noticed that after 100M frames, also using Rainbow, they report a better or similar performance to DeepEO in 4 out of 6 games (Asterix, Gravitar, Seaquest, and Space Invaders). So, could it be that everything that is being reported here is nothing but a consequence of temporally-extended exploration induced by options? This is obviously the most glaring concern, but there are others. For example, J-PER is cast as a contribution but it is never actually evaluated as such, I believe. The least I would expect is a comparison between J-PER and uniform sampling. Finally, on this topic, I don’t think the impact of ConstNeg was properly evaluated as well. I mean, one could see a uniform penalty throughout the state space as an optimistic initialization, which drives exploration (c.f. Domain-Independent Optimistic Initialization for Reinforcement Learning by Machado, Srinivasan, and Bowling). It is not necessarily bad. I wish there was some study about the impact of such a signal (or the generated option). Just casting it as bad to contrast it to a regular weighting of intrinsic rewards is dangerous.\n\nIn terms of overclaiming, I struggle to find how using multiple heads to predict the value of different cumulants (to use GVF nomenclature) can be seen as a contribution. This is a fairly standard approach to deep RL thus, is it really a contribution? Based on my previous comment on J-PER and this one, I believe the last two contributions stated in the paper are not justified. Maybe the first one is not as well, given the ez-greedy results.\n\nFor the method itself, it seems that both c_switch and w_s have a big impact on performance (Fig. 2). It is said that these parameters are easy to tune, but how can that be if there’s such variability across values close to each other? It is particularly tricky to balance loss functions, mainly when multiple intrinsic reward signals are considered. It seems to me that the proposed method has several parameters that might make it brittle. Importantly, operating at different time scales seem to be an important ability that we should not prevent our agents to have with a fixed c_switch value.\n\nFinally, besides what I already pointed out in the previous paragraph, the write-up itself is sometimes imprecise and some of the statements are not backed up. To name a few: \n- “explicitly looking for new knowledge and experiences”: I don’t know what knowledge is in this context, it is never defined. What is the difference to experience, for example?\n- “such as wasting world knowledge and transfer potential”: Is world knowledge different than “knowledge”? What is “transfer potential”? Was this evaluated in any experiment to justify this claim?\n- [Multi] “builds a stronger representation”. Is there any experiment to back this up? What is even a good representation?", "summary_of_the_paper": "This paper introduces an extension of Explore Options (EO; Bagot et al. 2020) to the non-linear function approximation case. EO consists in training two different policies in parallel, one to maximize the intrinsic reward, and one to maximize the environment's reward. Then, the agent can decide to act according to the option generated by the intrinsic reward for a fixed number of time steps, which provides it with temporally-extended exploration and bypasses the need to combine different reward signals by tuning a scalar value.\n\nThe method proposed in this paper extends EO by designing a neural network that has a shared torso and J different heads, one for each value being learned (environment's reward and intrinsic reward, for example). To do so, because it wants to use PER, it needs to adapt PER to deal with errors from multiple heads. It does so by keeping track of the errors in the two different heads and sampling transitions according to the J error values to ensure samples interesting for each of the J heads are present in the minibatch.\n\nEmpirical results show that the proposed method tend to outperform standard RND. This is relevant because the intrinsic reward function used to generate the exploratory option was the RND loss so, in a sense, this is a direct comparison between linearly combining reward signals and using them as an option.", "summary_of_the_review": "This paper introduces a new agent architecture that has several moving pieces and it doesn’t do a good job at evaluating their impact. In the end it is hard to claim that the value of the main idea (to use an intrinsic reward signal without having to linearly combine it to the extrinsic reward) was backed up with clear experiments. There are several confounders. For example, it could still be that the observed empirical performance is due to some sort of temporally-extended exploration, and that’s all. I don’t see data to rule this out. Finally, the paper makes several claims that are not precise or that are not supported by data, including some of its claimed contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LezwGdpPY5V", "reviewer": "Reviewer_svsx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n--\n- The main motivation of creating a method to incorporate multiple intrinsic reward signals is good\n- The proposed method makes intuitive sense -- simply learn separate policies from each reward signal off-policy\n- The proposed 2-PER method exhibits a performance improvement in the environment it was tested in\n\nWeaknesses\n--\n- W1 A variety of the claims are unsupported due to missing evidence or ambiguity in the experiments\n- W2 There are several important writing issues\n\nWeakness 1\n--\n- W1.1 It's odd to claim the method is J-PER, yet only apply it to the 2-PER setting. It's also not clear how to implement the proposed method beyond J=2 setting.\n- W1.2 The final set of intrinsic rewards used in the main method is unclear. Is it just RND and ConstNeg? Because of this ambiguity, it's also unclear whether the method's performance improves with the addition of more intrinsic rewards (a claimed contribution).\n- W1.3 The comparison is weak -- only two baseline exploration-bonussed methods are compared to. It would be good to include a skill-learning method. For instance [A] can be considered a method for learning intrinsically motivated options and then planning over them to achieve extrinsic rewards.\n- W1.4 The differences between the architectures (Shared, Separate, and Multi) are unclear. The main paper needs to be clearer about this point.\n- W1.5 The sweep of $w_s$ is only performed in one environment. This makes it hard to draw the conclusion that the proposed 2-PER algorithm is generally useful.\n\nWeakness 2\n--\n- W2.1 The related work section is 2 pages long, which is seems too long for a 9-page paper. I think its size could be significantly reduced.\n- W2.2 It's unclear why copying the buffer once for each agent is an important bottleneck.\n- W2.3 These losses ($\\mathcal L_e$ and $\\mathcal L_i$) have not yet been defined.\n- W2.4 The writing is generally informal and meandering. I recommend trying to make the writing much more concise. Otherwise, readers with less patience are likelier to ignore this work.\n\nMinor weaknesses\n--\n- Top of page 5: the use of an ellipsis here is too informal.\n\n[A] Sharma, Archit, et al. \"Dynamics-aware unsupervised discovery of skills.\" arXiv preprint arXiv:1907.01657 (2019).", "summary_of_the_paper": "This paper proposes an approach to learning intrinsically motivated options that is combined with an extrinsically trained policy. The motivation is to formulate a learning algorithm that can scale to the availability of multiple intrinsic reward signals better than an approach that attempts policy learning with a linear combination of extrinsic and intrinsic rewards. Experiments on several ALE environments illustrate that the method can perform better than a baseline method that incorporates an intrinsic reward with a weighted sum.", "summary_of_the_review": "While the motivation and method makes sense, many of the claims are unsubstantiated, weakly substantiated, or ambiguously substantiated (see Weakness 1). Because the focus of the paper is mainly on demonstrating the empirical utility of the proposed method, significant improvements are needed to make the paper acceptable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work gives an interesting perspective on combining options with exploration in the non-tabular case. The reviewers have raised a number of important areas for improvement (primarily missing ablations to support the claims of the paper, but also specific suggestions about improvements to the text), and feel that sufficient work is required to address these that the paper should be rejected at this time.
Decision: Reject