Deep Learning of Intrinsically Motivated Options in the Arcade Learning Environment
Authors: Louis Bagot
Kevin Mets
Tom De Schepper
Steven Latre
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
intrinsic motivation
exploration
options
auxiliary task learning

Abstract: In Reinforcement Learning, Intrinsic Motivation motivates directed behaviors through a wide range of reward-generating methods. Depending on the task and environment, these rewards can be useful, might complement each other, but can also break down entirely, as seen with the noisy TV problem for curiosity. We therefore argue that scalability and robustness, among others, are key desirable properties of a method to incorporate intrinsic rewards, which a simple weighted sum of reward lacks. In a tabular setting, Explore Options let the agent call an intrinsically motivated policy in order to learn from its trajectories. We introduce Deep Explore Options, revising Explore Options within the Deep Reinforcement Learning paradigm to tackle complex visual problems. Deep Explore Options can naturally learn from several unrelated intrinsic rewards, ignore harmful intrinsic rewards, learn to balance exploration, but also isolate exploitative and exploratory behaviors for independent usage. 
We test Deep Explore Options on hard and easy exploration games of the Atari Suite, following a benchmarking study to ensure fairness. Our empirical results show that they achieve similar results than weighted sum baselines, while maintaining their key properties.


Under review as a conference paper at ICLR 2023
DEEP LEARNING OF INTRINSICALLY MOTIVATED OP- TIONS IN THE ARCADE LEARNING ENVIRONMENT
Anonymous authors Paper under double-blind review
ABSTRACT
In Reinforcement Learning, Intrinsic Motivation motivates directed behaviors through a wide range of reward- generating methods. Depending on the task and environment, these rewards can be useful, might complement each other, but can also break down entirely, as seen with the noisy TV problem for curiosity. We therefore argue that scalability and robustness, among others, are key desirable properties of a method to incorporate intrinsic rewards, which a simple weighted sum of reward lacks. In tabular setting, Explore Options let the agent call an intrinsically motivated policy in order to learn from its trajectories. We introduce Deep Explore Options, revising Explore Options within the Deep Reinforcement Learning paradigm to tackle complex visual problems. Deep Explore Options can naturally learn from several unrelated intrinsic rewards, ignore harmful intrinsic rewards, learn to balance exploration, but also isolate exploitative and exploratory behaviors for independent usage. We test Deep Explore Options on hard and easy exploration games of the Atari Suite, following a benchmarking study to ensure fairness. Our empirical results show that they achieve similar results than weighted sum baselines, while maintaining their key properties.
INTRODUCTION
In Reinforcement Learning (RL), an agent is sequentially given states and needs to perform actions in order to maximize obtained extrinsic rewards r". The agent is therefore deeply tied to the reward signal, and tends to fail when said signal is sparse or noisy. When the environment is very complex or high-dimensional, it desirable for the agent to explore in directed way (Thrun 1992), i.e. explicitly looking for new knowledge and experiences. One of the most common ways to generate such task- independent, directed behaviors is through intrinsic motivation (IM), i.c. an alternative reward signal rf to spur curiosity and entice behavior exploration (Oudeyer & Kaplan 2009 Schmidhuber 2010).
IM biologically refers to the natural tendency of organisms to explore.
One of the most common benchmarks for Deep RL agents has been the Arcade Learning Environment (ALE, Bellemael et al. (2013)), consisting of Atari video-games, In order to solve the most chal- lenging, so-called hard-exploration games of the domain (Bellemare et al. (2016)), state-of-the-art Deep RL methods have integrated IM in complex learning mechanisms, and finally managed to overcome human-level play all 57 games (NGU and Agent57, Badia et al. (2020b a)). However, these methods still fundamentally rely on a Weighted Sum (WS) of rewards Tr = rf + Bri, for very well-chosen and complex IM reward. So while the types of behaviors that we can extract with IM keep on expanding, we are still ultimately relying on a single signal to help exploitation. Instead, we might want to benefit from different and complementary intrinsic signals (Matusch al. [2020). In this paper, we refer to this challenge as IM Incorporation (IM-Inc). We extract several key desirable features of IM-Ine methods, including scalability, robustness and generality.
In a tabular setting, Explore Options (EO, Bagot et al. (2020)) have been proposed as alternative to the a weighted sum of rewards. The agent is divided into the Exploiter, trained exclusively with the extrinsic reward, and the Explorers, trained exclusively with the intrinsic rewards. The Exploiter can call any Explorers through options (Sutton et al. 1999), i.e. additional actions, to explore for a fixed amount of time. However, because of their tabular nature, Explore Options are inadequate for function approximation; but it is a crucial element to allow the agent to generalize the option over states and effectively learn to balance exploration. We revise Explore Options into Deep Explore

---

Under review as a conference paper at ICLR 2023
Options (DeepEOs), new method for combining intrinsic and extrinsic reward signals in Deep RL. We introduce several key changes to usual IM approaches, and showcase their performance in the ALE. To provide fair and controlled comparisons, in benchmarking study (benchmar k. TTigaa et al. 2019)) on IM.
match the algorithm and hyperparameters used
The contribution of our work is fourfold:
We revise Explore Options within Deep RL to propose Deep Explore Options as a strong alternative to weighted sum when using intrinsic rewards, extending the ben.cmarrk.
We empirically show that, unlike a weighted sum of rewards, DeepEOs can learn from several IM rewards at once, and ignore harmful signals.
We empirically show that DeepEOs can extract the exploiting behavior, while learning meaningful and potentially transferable exploratory behaviors.
We provide a study of methods to combine intrinsic and extrinsic rewards. We propose several key desirable properties of such methods, and place several existing works within this framework, including DeepEOs.
In Section 2 we provide background to build Deep Explore Options. Next, we introduce our method and discuss the introduced elements Section In Section 4 we provide experiments in MiniGrid to build intuition, then in Atari following the benchmark rk. We go over existing work in the field in Section [S] Finally, in Section [6 we provide a study of methods to combine intrinsic and extrinsic rewards.
2 BACKGROUND AND EXPLORE OPTIONS
2.1 REINFORCEMENT LEARNING, OPTIONS, MOTIVATION
We use the standard RL setting (Sttonn& & Barto 2018 modelling the environment Markov Decision Process (S,A, R, p, 7) where S is the set of states, A is the set of actions, R C IR is the set of rewards, p A, S, R + [0, 1] is the dynamics function, and y is the discount factor. The goal of RL is to maximize the expected sum of discounted rewards from any starting state.
Options (Sutton et al. (1999) ) refer to temporally extended actions. An option defined as a triple (I, 3, 3), where I C S is the option's 's initiation set, i.c. states in which the option can initiate; π is the option's policy; and S + [0, is the option's 's termination condition.
We assume one or several intrinsic reward functions f (s, a, s') r to generate a reward that we are interested in learning from. These can attempt to motivate directed exploration behaviors, but also any other behavior in the environment. An overview of the literature populating the field can be found in Section 5
2.2 EXPLORE OPTIONS
Explore Options (EO, Bagot et al. (2020) have been introduced as an alternative to weighted sum of rewards. The method consists decoupling the Agent into an Explorer, trained with the intrinsic reward r', and an Exploiter, trained with extrinsic reward rf. Switching from Exploiter to Explorer is done through the Explore Option, which the Exploiter can use at any time to let the Explorer act for fixed amount of steps Cswitch. Within the options framework, the jth Explore Option oj is therefore defined as (S, TT. Seaweess)) ). where the initiation set the entire state space, the option policy πy is the Explorer policy trained with intrinsic reward y².._ and the termination function Beswiten, deterministically interrupts the option call after Cswittch steps. Explore Options have only been introduced a tabular setting. However, by design they only make sense in a function approximation setting: the option learning requires generalization in order to call the option in states where little is known, and therefore directed exploration is required. Function approximation also allows for parameter sharing, and potentially generalization across tasks.
3 DEEP EXPLORE OPTIONS
Fig. gives an overview of the general Deep Explore Option framework, of which we go into more detail in the following subsections.
2

---

Under review as a conference paper at ICLR 2023
Stal fe1
CUV
C3V
Replay Butler
Mini-batch
S,G, 5,
S241
ranomm samphngg
learting
301
D
Deep Explore Option framework. During interactions, the state is passed through the DeepEO agent and either an Explorer (green) or the Exploiter acts according to the Explore Option. All transitions, enhanced with intrinsic rewards, are stored in the replay buffer, and sampled uniformly into a mini-batch, regardless of which agent had gathered them. The learning is therefore done off- off-policy all transitions of the buffer.
Figure 1:
3.1 CORE DIFFERENCES WITH USUAL INTRINSICALLY MOTIVATED AGENTS
Option Learning A weighted sum of rewards relies on the IM rewards themselves guiding dual exploiting-explorngg agent towards relevant parts of the space. Instead, our IM is used exclusively to train an independent agent, that might be called through an Explore Option. We therefore rely on off-policy learning, i.e. the Explorer showing interesting transitions to the Exploiter, rather than merging the two behaviors avoiding a complex / tuning. Using the option has the benefit that the Exploiter can learn when to explore, e.g. default behavior when rewards are known.
Multiple Intrinsic Rewards Intrinsic reward functions frr are often based on heuristics and intuition regarding what signal would motivate the agent to produce the most interesting and beneficial behaviors. The range of possible behaviors produced by Intrinsic Motivation is as wide as the task space, and widens with the complexity of the environment. Therefore, as we will further discuss in Section 6 and recommended by Matusch et al. (2020), it is desirable to learn from several intrinsic single agent per reward in DeepEOs, any off-policy algorithm
reward signals. Due to the learning guarantees the optimization of the intrinsic reward, and thus ensures scalability. The only practical issue with such approaches that learn different agents is if the scarce relative interacting of all agents begets the necessity of Offline RL (Fujimoto et al. 2019 Levine al. 2020
Sharing parameters of the model with auxiliary tasks in RL has been shown to provide a denser learning signal and accelerate learning. by building an enviromment-awar representation (Jaderberg et al., 2017: Lample & Chaplot, 2017). However, the learnt prediction and control models are rarely used beyond enriching the loss. We consider the learning of a fully intrinsically motivated policy value function as an auxiliary task, but still use the learnt policy in practice. We therefore construct a feature representation that we branch out into our different Agent heads (see A.2 for the precise Architecture). The now-shared visual representation therefore needs to encapsulate relevant elements for both our Explorers and Exploiter. This type of architecture is quite common when learning from multiple signals (Barreto et al. 2018), but very rarely so when dealing with IM, mainly because the WS hard to scale. In our experiments, a shared architecture is 30% lighter and faster than separate networks. Note that each head and independent parts of the same optimizer can adjust for their own reward scale. In addition, our meatie architecture allows for efficient swapping, isolation or resetting of control heads, meaning that adding an Explorer or switching tasks can be done while preserving both the Explorer behaviors and the visual representation. We combine the losses through C L + X Σ, C'j, with 1, which required no tuning. À not be confused with the 3 from Ws, since each head adjusts for their own reward scales, meaning that A does not carry the heavy duty of reward balancing.
Auxiliary Tasks
3.2 BACKBONE & TRAINING
Benchmarking Study Experimental protocols in IM tend to be chaotic, with new approaches usually introduced on arbitrary environments and algorithms, and only compared to non-IM baseline. Taiga et (2119) (benchmark) provides an experimental setup to fairly compare IM methods in the ALE, using the WS scheme. It uses the Dopamine framework (Castro al. 2018).
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Agent Loss Computation
Input: Buffer 13, Exploiter Q", J Explorers Q and associated reward functions f , Vjj
Uniformly sample batch (s,a, r", r" ,r .. , dome) ~ B // may contain in opt ons 11 mask of actions that are options
option_man _aac = a in options
L compute loss sQ
done)
for j in JJ do
L = compute_loss loss (firs; s, s') #option_mask
// mask out options
L compute loss (Q");8, ;8,a r") , done= = False) toplion toplion_mask
L= L + +, L3 + E, (
Output: Total Loss C
which provides an implementation of the Rainbow agent (Hessel ct al. 2018) with its 3 majors improvements on DQN (Mnih et al. 2015): Prioritized Experience Replay (Schaul et al. (2016). n-step returns, and Categorical DQN (C5I, Bellemare et al. (2017). We match this setting and all hyperparameters using the DeepRL PyTorch implementation (Zhang 2018).
Intrinsic Reward Functions The benchmark compares several intrinsic reward functions; for our first IM function we use Random Network Distillation (RND, Burda et al. (2019))) as it the simplest, most used method, and is currently part of state-of-the-arr methods (Badia et al. 2020b a). RND uses a fixed, randomly initialized target network f S > 1='' and a learnt predictor network f:S + Rm.. The method generates an intrinsic reward n² = fürl (s, a, s') (I') - (s') 1 by distilling the target into the predictor, using the loss motivation signal. This will therefore lead the agent towards less-visited states, that still show a high distillation error. In order to keep our return estimates within C51's usual [-10, 10] range, we scale the RND reward by 7) = 0.01. This is not to be confused with the / from WS, as this is vastly easier to tune: it is only meant to keep the RND returns within a reasonable scale and never meant to balance different reward signals.
In order to study a multi-IM setting, we use another intrinsic function that is trivial to implement: Optimistic Initialization (Machado et al. 2015). It can be obtained through a simple constant negative reward, which will push the agent off visited grounds. The original paper balances for the lost rewards at termination, but our purely intrinsically motivated agent is not provided terminal signal, and can therefore simply be fed a constant n22 -0.025. In this scenario, this agent is a close cousin of the DORA method of exploration (Fox et al. 2018).
Since virtually all IM rewards are based on heuristics, they are bound to break in certain scenarios and environments. A common example of such failure is the noisy TV problem (Burda et al. 2019a):
an agent trying to find sources of surprise will be mesmerized by unpredictable noise. In order to test our method for such a breakdown, we implement a third intrinsic reward function pr ~ ~ 0O 1) that samples rewards randomly in each mini-batch. An agent trained with this will be chaotically attracted to random parts of the state-action space.
Agents and Option Training We build our multi-headed agent with Exploiter head and several Explorer heads. We add actions to the Exploiter, corresponding to the EOS.. When the option oj is called, the corresponding Explorer j acts for fixed Cswitch steps before giving control back to the Exploiter. This transition (s, a, r, s') (st, 0j, R, St+sowers)) ) with Cawiton 2/rr rf+ is added to a unique buffer along the rest of the n-step transitions, regardless of the current actor. We train all agents simultaneously. ofllpy, using the same uniformly sampled buffer data, masking the option transitions for the Explorers, as explicited in Algorithm []] The Explorers are not provided the done signal, since it might leak information about the task, rush agents with negative rewards, and interfere with exploration (Burda al. 2019a).
Ei=o
In the Dopamine implementation of Rainbow, e-greedy is the main strategy used for exploration. Keeping this unchanged for options would mean enforcing Cewitoh-long exploration phases when the options are randomly called, leading to much more exploration than e-greedy intends. Instead, we note that e-greedy means allocating fraction of the agent's 's time exploring. In order to preserve this idea, we enforce our Explorer agents to occupy c/2 of the agent's 's time, while our Exploiter otherwise follows e/2-greedy exploration strategy.
4

---

Under review as a conference paper at ICLR 2023
10-
OogeDa
DiepCCccrry Ost
DispEO with Aucilayy Tasls
Secarate Exolober
Siparate Explore
DON--B
DON
2.-
(b) Visual intuition for DeepEOss: [top left] early policy, after 250k steps, where E represents a DeepEO call (blue: goal, agent orientation: down); [top right) an Explorer trajectory: [bottom] Learnt features with and without auxiliary task learning, when following the Explorer trajectory.
30-
0.90
0.50
(a) Ablation Study of DeepEOs in Roor-Room Minigrid, and comparison with DQN DQN+IM baselines.
Figure 2: Experiments in MiniGrid.
EXPERIMENTS
4.1 MINIGRID INTUITION
We start by building intuition for DeepEOs in MiniGrid (Chevalier-Boisvert et al. (2018)) 2D visual gridworld. We use a classic four-room environment with actions (forward, left, right) and oully-observable inputs. We implement simple first-cell-visit intrinsic reward fir (s, s')
e'gr.
which we use to train our DeepEO agent based on the DQN algorithm (Mnih et al. (2015) ). We also use it to train a weighted sum DQN+IM agent baseline, along with untouched DQN agent. We design an ablation study of DeepEOs, removing either the Option or the Auxiliary Task learning (Section 3.1 leave only the other. The results can be found in Figure 2a We without ablations, DeepEOs perform better than the baselines. The ablation studies show that both the option and the auxiliary tasks are beneficial for learning in this scenario, with the auxiliary task rather improving early performance, and option usage rather late performance. In Figure 26 we visualize several key components of the method. First, we show the Exploiter policy early training, after 250k steps. We can see that the agent quickly learns to call the Explorer (action E) in states distant from the goal. Next, we show an Explorer trajectory towards the end of training, confirm that achieves strong and independent covering policy. Finally, we study the representation with and without auxiliary task learning, by observing the network's 's latent representation during Explorer trajectory. We freeze the agent's position in the trajectory We can see that the Exploiter alone (similar to a DQN agent) essentially ignores the coverage map. However, the Explorer needs it to act, its features are sensitive to it. Combining both networks therefore forces the Exploiter's 's representation to contain information about the coverage map, which useful for performance (Figure 2ah. More information about all experiments in this subsection can be found in Appendix A.
that
only let the coverage map change.
4.2 ATARI EXPERIMENTAL PROTOCOT.
We now evaluate Deep Explore Options on the Arcade Learning Environment, with RND and Opti- mistic Initialization Explorers, with the same games, core algorithm, architecture and hyperparameters as the Rainbow implementation from the benchmark. Architecture and full experimental details can be found in Appendix A.2 We use two baselines provided in the benchmark: Rainbow with e-greedy (Rainbow) and Rainbow with an RND intrinsic reward using WS (Rainbow+BRe refer to our method as DeepEC. In order to test our exploitation behavior, we evaluate our Exploiter under e-greedy without options every million frames; referred to Expl Exploiter.
We
5

---

Under review as a conference paper at ICLR 2023
Draw Lar
Mo Mecemaaeeeeegg
Attert
Rainbow
Rainbow+
DeepEO
Exploiter
DeepEO Random
Mileen ot
Salurt
Spoce
solr
Milliu
PVDDE
Freeway
Millrr of
Millow
Villine
Figure 3: Experiments on the Atari Suite. Two left columns: hard-exploration games. Two right columns: easy-exploration games. The blue curves represents the c-greedy performance of the Exploiter without forced usage of the Explore Options. The orange curve represents the DeepEO in robustness challenge: with a purely Random reward.
4.3
MAIN RESULTS
Performance and Scalability Our main empirical results are shown in Figure 3 We observe that our results are overall competitive with the baselines. Except for SOLARIS, the Exploiter always improves on DeepEO where option usage is forced. On PRIVATEEYE, our agents learns sooner, but therefore also unlearns sooner this unlearming is common in this environment. The two most notable outliers are GRAVITAR, where DeepEOs do not manage to match Rainbow, and MONTEZUMA, where it does not manage to match Rainbow+RND, We address the latter in 4.4 but note that we always improve on the lower bound. In FREEWAY but most notably in PITFALL, the DeepRO performance lags behind the Explot oollloteen ter. While this is expected in Freeway, since there is always small chance to call the Explorers for Cewitch steps, the case of Pitfallll is interesting: the Exploiter wants to do nothing to get return of O, but both Explorers are attracted interactions that eventually lead to negative rewards. The Exploited successfully ignores them. This is a first hint that the Exploiter ignores harmful behaviors and extracts the exploiting behavior.
Robustness Following this remark on Pitfall, we study Robustness by introducing our third IM reward: the purely random R2 ~ N (0, 1), simulating signal that degenerates. This is represented as DEEPEO+RANDON in the tuning games. We can see Figure 3 again that this new intrinsic reward had absolutely no harmful consequence on our performance. matching DeepEO without a hitch. This shows that DeepEOs are perfectly robust to destructive IM rewards. For the sake of argument, we try Weighted Sum of our rewards and report the results Appendix B 6 as MULTI RAINBOW. As expected, weighted sum of rewards is not scalable robust unless under heavy $3 fine-tuning.
Additional Experiments We include two more experiments in Appendix B first, a study on the Cowiitch hyperparameter, where we find performance be very robust to its changes. We choose Cewitch = 50 in the experiments we display in the main text as the best value by a small margin. Next, we study naive PER prioritization for 2 agents, which find does not benefit the agent; therefore used random sampling for our DeepEO agent. We encourage future work to design algorithms to perform multi-agent prioritization in similar context.

---

Under review as a conference paper at ICLR 2023
Figure 4: Visualization of some learnt behaviors of, respectively, the RND and Optimistic Explorers. Red to green respectively mean older to newer visits in a single episode. On MONTEZUMA (left), both agents learnt to reach the key, but the RND Explorer is much faster, while the Optimistic Explorer tends to jump more but lags on the ladder leading to the key. On FREEWAY (right), the RND Explorer interested by the fast-lane cars in the middle, while the Optimistic Explorer likes the upper, less visited section.
4.4 LEARNT BEHAVIORS
Previous research has shown that temporally extended exploration can be surprisingly efficient exploration method (Nachum et al. 2019 Dabney et al. 2021), even when said exploration is random or spamming. While this motivation for our work, it raises an important question did our Explorers learn any relevant behaviors, or are we simply benefiting from observing several steps of random off-policy behavior? While Figure 2b gave a positive answer for MiniGrid, we now study Atari. We visualize the learnt behaviors in Figure the episode, while greener represent most recent timesteps. In MONTEZUMA, both the RND and Optimistic Explorers learn to reach the key on their own, showcasing long-term planning. We note the relative weakness of the independent policies compared to a full online PPO RNN agent in the original RND paper. We believe improving the Explorer quality is important axis for future improvement, and recommend studies on the types of algorithms that suit IM most, and on the impact of near-offline learning. These results are still motivating for Task Transfer: in Campos al. [202]). the authors pre-train an exploratory behavior and use it as an option downstream. As they mention, this approach complements well with EOs, as they allow training of the exploration policies in tandem with exploitation.
Redder shades indicate earlier timesteps in
RELATED WORK
Exploration without IM Methods that do not incorporate IM were state-of-the-ar on Atari until recently. Such methods generally injected noise in the action or parameter space to explore. In Value-based methods, e-greedy (Mnih et al., 2015) randomly explores with some probability each step, while Noisy Nets (Fortunato et al., 2018] 2018)HHsse Hessel et al. (2018) introduces learnt noise in the parameters. Policy-based methods, entropy maximization (Haarnoja et al., 2018) takes advantage of the stochastic policy formulation to motivate noisier policies.
Knowledge Acquisition and Skill Learning IM (Aubret et al. 2019) is generally used for knowledge acquisition about the environment. Motivating exploration is by far the most common goal then, and divided into state novelty (Bellemare et al.) 2016)) Ostrovski et
Intrinsic Motivation:
2017
Burda et al., 2019b),, prediction error (Pathak et al. 2017) and information gain (Schmidhuber
2010). Beyond knowledge acquisition, IM can also encompass skill learning, where the agent learns a actions to help solve the task, sometimes called option- or skill-discovery: The main body work involves reaching states with key properties, for example through Proto- Value Functions Machado et al. (2017). algebraic graph connectivity (Jinnai et al., 2019a b). the Successor Representation (Ramesh et al. 2019) or simply a measure of diversity (Eysenbach et al. 2019). These methods generally attempt find a coherent set of options, they do not try incorporate other forms of IM exploration. In other words, Explore Options generalize these ideas to any set of intrinsically- mottivated options, and could benefit from these methods as Explorers.
set of cemporalll--xtenndee
7

---

Under review as a conference paper at ICLR 2023
6 DISCUSSION: COMBINING REWARD SIGNALS
As explored in the previous section, the generation of useful, auxiliary reward signals under the form of intrinsic motivation has been active subject in recent literature; refer to this task as IM Generation (IM-Gen). However, addressing how to incorporate and learn from these reward signals has seen surprisingly less attention; we refer to this task as IM Incorporation (IM-Inc). In this section, we extract key desirable properties of IM-Inc methods. We then discuss the position of several sfate-of-the art methods in this framework.
6.1 DESIRABLE PROPERTIES
We start by assuming the usage of several IM-Gen methods firg (s,a,s') R..j. as set of rewards that simply might be useful. The goal in introducing these is to boost the present or future exploiting performance. By fulure, we mean to cover Transfer and Lifelong Learning: learning representations and behaviors that will solve downstream tasks in similar environments. With this in mind, we find the following desirable properties for IM-Inc method:
Scalability is not realistic to expect of a unique IM function that it should
maximally assist the exploiting behavior. IM-Gen methods generate all types of behaviors and data, and might complement each-other within a single task or over tasks. From Matusch et al. (2020): "we find that input entropy and information gain are similar objectives while empowerment may offer complementary benefits, and thus recommend future work on combining intrinsic objectives." In Ramakrishnan et al. (202T), the strongest exploration-bas based rewards change drastically with the environment and task. An IM-Inc method should therefore be able to combine several IM rewards.
Robustness Following scalability, among the many rewards, some will be less effective, while some might even degenerate for portions of or entire tasks (e.g. noisy TV problem, Burda et al. (2019)). An IM-Inc method should be able remain unaffected by harmful signals.
Behavior Assimilation IM, in essence, leads to a conflict of interest between exploitation and the IM behaviors. An IM-Inc method should therefore manage assimilate these behaviors in way that helps exploitation in all scenarios, and does not risk losing any of the relevant behaviors. A simple degenerating scenario is that an r ggrab + Trun agent could lose both independent behaviors to chaos.
Generality To ensure convergence, IM-Gen methods often rely on some assumption on the rewards for example that they will disappear in time, or that they cover exploiting behavior. However, this is vastly limiting in the types of rewards and behaviors it allows. Therefore, along with scalability, should be effortless to introduce and swap in new IM-Gen rewards, regardless of their nature. An IM-Inc method should be minimally restrictive on the rewards it allows, and let them swap in easily.
Transferability of Behaviors Most IM-Gen methods produce independent behaviors that are meaningful for several tasks. Learning such behaviors is therefore of great interest not only in the current task, but also for future ones, in task transfer scenarios under similar dynamics. The ability to preserve and re-use these behaviors is thus of great relevance to severely boost the learning of down-stream tasks, as done in Campos et al. (202T). We refer to this idea as Behavior Transfer.
Representation Learning In a function approximation scenario, the learning of an IM policy always requires understand the structure of the state and MDP to some extent, through some form of representation learning. Having the exploiting behavior benefit from this understanding through auxiliary tasks or additional inputs can strongly benefit performance. Even beyond this, and following transfcrability, the ability to extract and re-use these learnt representation for down-stream tasks could lead to substantial gains performance (e.g. Gordon et al. (2019). We refer to this recyeling idea as Representation Transfer (Lazaric 2012).
6.2 STUDY AND TABLE
We identify several methods that can categorize as IM-Inc from the literature. We organize them in Table followingg the key properties we have formulated.
First and most obviously, a weighted sum of rewards, as done in the benchmark, is the most common IM-Ine method. The behavior assimilation method is through merged exploiting/epplorttry Agent that requires fine attention to 8, but this makes it hard to scale and hard to swap in new rewards. It is also very restricting: if the intrinsic reward doesn't disappear in time, it will always harm the
8

---

Under review as a conference paper at ICLR 2023
Method
Scalability
Robustness
Behavior
Generality
Transfer Potential
Representation
Assimil.
Learning
Weighted Sum
No
No
Merged
No
No
Merged
NGU
No
Yes
Merged
No
No
UVFA
Agent57
No
Yes
Merged
No
Yes
No
Successor Features
Yes
Yes
GPI
Yes
Aux. Tasks
Explore&Exploit
Yes
No
Separate
Yes
Yes
No
DeepEO
Yes
Yes
Separate
Yes
Yes
Aux. Task
Table 1: Comparison of methods to combine intrinsic and extrinsic rewards
exploiting behavior. Never Give Up (NGU. [Badia et al. (2020b) ) combines intra and inter-episodic
IM reward through a complex mix of scaling, bounding and eventual weighted sum: we see this reward the limit of single-reward formulations. In NGU, UVFA network (Schaul et al. 2015) allows to pass 3 as input and thus fine-tune 3 during training-time, This makes NGU robust to harmful signals. While the UVFA is beneficial for Representation Learning, it does not allow for efficient Representation Transfer since we cannot untangle the Exploiter and Explorer to isolate the second, Agent57 (Badia et al. 2020a) improves on NGU by further tine-tuning with a Multi-Armed Bandit, and replacing the UVFA by decoupled Explorer and Exploiter Value Functions for stability. This effectively enables Behavior Transfer of the learnt Explorer to degree, but the Exploiter does not benefit from the Explorer representation anymore.
Successor Features (SF, originally Dayan (993)) are a family of methods relevant here through the General Policy Improvement Theorem (GPI, Barreto et al. (2017) and Barreto et al. (2018) allowing to extract the strongest available policy from any set of base policies. While this is ideal for Behavior Assimilation and Transfer, current SF methods require very expressive set of base rewards in order to formulate any new reward as a linear combination of the basis. This assumption is yet to be proven viable in general setting. Regarding Representation learning, |Barrtto et al. (2018) used an architecture similar to ours, sharing parameters in an auxiliary task manner.
Finally, include with DeepEO method along with its closest cousin, Explore&Exploit (Nachum iet al. 2019). Explore&Expii independently trains purely IM Agent and samples the next Agent between Explorer and Exploiter every Cawitch steps. Due this fixed sampling, this method is robus while DeepEOs can learn to ignore bad options (Section 4.3) Both methods are scalable and general (4.3), as guaranteed by the learning of a separate behavior per reward function (Section 3.1 Note that using separate behaviors is necessarily weaker than the GPI, by definition. The shared architecture of DeepEOs allows to benefit from the Explorer Representation (Section 4.1). Regarding Transfer, Campos et al. (202T) provide a method to efficiently transfer from learnt exploratory behaviors using Cawstch = I for Exploiter-called Explore Options and a higher value for random option calls. They mention Explore Options as complementary method to their work for training the Explorers in tandem with the Exploiter.
7 CONCLUSION
We have introduced a taxonomy for methods to combine intrinsic and extrinsic rewards in Reinforce- ment Learning (IM-Inc). We have extracted, desirable properties of such methods: scalability, robustness, behavior assimilation, generality, and transferability of the behavior and representation. Most of these are not met by weighted sum of rewards, the usual approach combining rewards. We have revised Explore Options to Deep Explore Options, to call an intrinsically motivated agent through an option in function approximation setting. To ensure fair comparisons, we followed benchmarking study that uses hard- and easy-exploration games of the Arcade Learning Environment. We showed here that Deep Explore Options are a strong alternative with similar performances on most games. But in addition to performance, we have empirically demonstrated that DeepEOs possess most of the desirable properties of an IM-Inc method in particular scalability, robustness, generality and representation learning.
weighted sum of rewards,
We should note that Explore&Fxxloit isn't meant as an IM-Inc method, but study Hierarchical RI..
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Arthur Aubret, Lactitia Matignon, and Salima Hassas. A survey on intrinsic motivation in reinforce- ment learning. arXiv preprint arXiv: 1908. 069760 2019.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR, 2020a.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020b.
Louis Bagot, Kevin Mets, and Steven Latré. Learning intrinsically motivated options to stimulate policy exploration. Fourth Lifelong Machine Learning Workshop at International Conference on Machine Learning, 2020.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501- 510. PMLR, 2018
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Conference Neural Information Processing Systems, pp. 1471-1479, 2016.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. International Conference on Machine Learning, pp. 449-458. PMLR, 2017.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large- scale study of curiosity-driven learning. In International Conference on Learning Representations, 2019a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019b.
Víctor Campos, Pablo Sprechmann, Steven Hansen, Andre Barreto, Steven Kapturowski, Alex Vitvitskyi, Adria Puigdomenech Badia, and Charles Blundell. Beyond fine-tuning: Transferring behavior in reinforcement learning. arXiv preprint arXiv:2102. 13515, 2021.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018.
Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium.
http://ocooaaaaaaaaaaaaaaaaaaaae Foundat htt.a///oo....otooooooooooooooon
2018.
Will Dabney, Georg Ostrovski, and Andre Barreto. Temporally-extendee E-greedy exploration. In International Conference on Learning Representations, 2021.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation.
Neural Computation, 5(4):613-624, 1993.
Benjamin Eysenbach. Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2023
Meire Fortumato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference Learning Representations, 2018.
Lior Fox, Leshem Choshen, and Yonatan Loewenstein. DORA the explorer: Directed outreaching reinforcement action-selection. In International Conference on Learning Representations, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062. PMLR, 2019.
Daniel Gordon, Abhishek Kadian, Devi Parikh, Judy Hoffman, and Dhruv Batra. Splitnet: Sim2sim and task2task transfer for embodied visual navigation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 1022-1031,2000
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference Learning Representations, 2017.
Yuu Jinnai, Jee Won Park, David Abel, and George Konidaris. Discovering options for exploration by minimizing cover time. In International Conference on Machine Learning, pp. 3130-3139, PMLR, 2019a.
Yuu Jinnai, Jee Won Park, Marlos C Machado, and George Konidaris. Exploration in reinforcement learning with deep covering options. In International Conference on Learning Representations, 2019b.
Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning. In AAAI Conference on Artificial Intelligence, volume 31, 2017.
Alessandro Lazaric. Transfer in reinforcement learning: a framework and a survey. In Reinforcement Learning, pp. 143-173. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643 2020.
Marlos C Machado, Sriram Srinivasan, and Michael Bowling. Domain-independent optimistic initialization for reinforcement learning. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Marlos C Machado, Mare Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. In International Conference on Machine Learning, pp. 2295-2304. PMLR, 2017.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Brendon Matusch, Jimmy Ba, and Danijar Hafner. Evaluating agents without rewards. arXiv preprint arXiv:2012. 1133,, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
11

---

Under review as a conference paper at ICLR 2023
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarehy (sometimes) work so well in reinforcement learning? CoRR, abs/1909.10618, 2019.
Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In International Conference on Machine Learning, pp. 2721-2730. 2017.
Pierre Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 16-17,2017.
Santhosh K Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman. An exploration of embodied visual exploration. International Journal of Computer Vision, 12965)16116-169, 2021.
Rahul Ramesh, Manan Tomar, and Balaraman Ravindran. Successor options: An option discovery framework for reinforcement learning. arXiv preprini 2011,
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, pp. 1312-1320, 2015.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference Learning Representations, 2016.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010), IEEE Transactions Autonomous Mental Development, 2(3):230- 247, 2010.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
Richard S. Sutton, Doina Precup, and Satinder P. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 11((1-2)181-211, 1999.
Adrien Ali Taiga, William Fedus, Marlos C Machado, Aaron Courville, and Mare G Bellemare. On bonus based exploration methods in the arcade learning environment. In International Conference on Learning Representutions, 2019.
Sebastian Thrun Efficient exploration in reinforcement learning. Technical Report CMU-CS-92-102, Carnegie Mellon University, Pittsburgh, PA, January 1992.
Shangtong Zhang. Modularized implementation of deep rl algorithms in pytorch. https:// g t hub. hwwp///Saanggtoggggagn//eeeeeee 2018
12

---

Under review as a conference paper at ICLR 2023
State maps the trajectory, averaged steps
Figure 5: Minigrid: One-hot encoded maps passed the agent, here averaged over covering
trajectory, keeping the position fixed. In order: [walls, agent agent y, visits, goall, goal2].
A IMPLEMENTATION & EXPERIMENTAL SETUPS
A.1 MINIGRID EXPERIMENTS
For simplicity, the agent is passed fully observable tensor of its state in the environment. In particular, to avoid requiring a memory, the state we use for Minigrid experiments contains one-hot encoded maps as [walls, agent_x, x, agent_y, visits, goall, goal2]. The agent and y correspond its orientation in space, only passed at the cell the agent is occupying. The visit map marks as each visited cell. The agent needs to ignore the last goal2 map.
For our feature-visualizze visualization experiment, we used trajectory of 50 steps (top right of Figure 2b and fixed the agent positions maps, leading Figure 5 over the trajectory.
only the visitation map changes, here average
We trained our DQN-based agents for 2 million steps, with 50k steps of random behavior, decreasing from to 0.01 over 100k steps, network updates every 4 steps, target network updates every 10k network updates (40k steps), learning rate 0.00025 with Adam optimizer, discount of 7 = 0.9. The DQN+IM agent fine-tuned to 0.1 as the best hyperparameter value. Our DeepEO agent used Cawstch = 50.
A.2 ATARI HYPERPARAMETERS & ARCHITECTURES
Hyperparameter
Value
Discount factor y
0.99
Min history to start learning
80K frames 32K frames
Target network update period
Adam learning rate
6.25 5
Adam
1.5
Multi-step returns n
Distributional atoms
51
Distributional min/max values c-greedy schedule
[- 10, 10]
0.01 over 1M frames
Table 2: List of Rainbow hyperparameters used, taken from the benchmark
Hyperparameters
The list of used Rainbow hyperparameters can be seen in Table 2 above.
The network architecture used based on the Categorical DQN, as used in Dopamine (Rainbow without Dueling). The network has Convolutional layers followed by fully-connected layer, which we refer to as the "vision module". followed by fully-connected output layer, which call "control module". The Exploiter has one more action per Explorer. Each Explorer attaches "control module" to the shared body "vision module".
Architecture
Following the benchma rk and recommended practices (Machado et al. 2018), < 0.25 sticky-actions and no termination life loss.
Experimental setup
Agents are trained over 5 seeds for up to 100 million frames on the 11 games selected by the benchma rk. 6 of them are hard-exploration games, while 5 are easy-exploration games.
13

---

Under review as a conference paper at ICLR 2023
parfummmac perfrmmncca
C-12
Sanbow
Scone (Pa Naaintow part
Gnviar
Figure Additional experiments: Csmitch study; Prioritization; multi-reward Weighted Sum
B ADDITIONAL RESULTS
B. Cowitcch AND PRIORITIZATION STUDIES
We perform experiments on our three tuning games: GRAVITAR, MONTEZUMA and ASTERIX. We chose these because they are mix or easy and hard-exploration that provide relatively smooth improvements with better methods. We could not match the set from the benchmark as it tunes on games outside the main set of 11.
Camritch hyperparameter In Bagot et al. (2020), the authors linked EO's Cawiitoh hyperparameter WS' B, since both dictate the intensity of exploration in their respective approach. It shown that Cawstch was much more robust and lead to overall better performances than any value of 3 in simple setting. We do have access to the benchmark per-ó performance but only the range: (0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001, 0.0005, 0.00001, 0.000005). We replicate this study Atari by studying Cawitch E (15, 50, 200). The results are available Figure 6 normalized by the Ra i nbow performance. We find that Cswitch does not have a lot of impact in performance, with tuning games. Cawstch 50 did better on GRAVITAR and ASTERIX by more convincing margin than Cawitch = 200 did MONTEZUMA. We therefore pick 50 as our main hyperparameter value.
all values leading to very similar performances
Prioritization Rainbow uses Prioritized Experience Replay (PER, Schaul al. (2016)) to sample more interesting transitions from the buffer, where the notion of priority of a transition is its associated Rainbow loss. Since we now have least agents, and therefore two notions of priority, a naive adaptation of PER would lead to each agent sampling We implement this idea, but can be seen Figure 6 random sampling on both GRAVITAR and ASTERIX the only one on which it was quite beneficial unsurprisingly being MONTEZUMA. potential cause for this gap performance could reside the option transitions: their higher average loss scales might be interfering with the sampling process. We leave deeper investigations of such objti-objective prioritization to future work.
according to its own preferences. we find that vastly under-pent sedder-perfoms against
14


Reviews:
[{"review_id": "s5IrXsqOYu8", "reviewer": "Reviewer_aMNX", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the use of previously proposed Explore Options to deep Reinforcement Learning (RL). It tests different methods to incorporate intrinsic rewards, that go beyond the typically used approach of summing extrinsic and intrinsic reward signals.", "summary_of_the_review": "The paper has major weaknesses counterbalanced by a few strengths. The paper claims that the proposed contribution is \"fourfold\"; this is concerning to me, since in my opinion the contribution of this paper is actually quite minimal. I recommend the author seriously reconsider this work and resubmit it only with substantial improvements.", "strength_and_weaknesses": "This paper proposes a basic deep RL extension of previously proposed Explore Options. This extension is very straightforward, as evinced by the very small description dedicated to it in the paper, and it does not have scientific significance or novelty. The methodological section investigates ways to combine previously introduced techniques to decouple the learning of intrinsic and extrinsic rewards, without any theoretical intuition of how these methods should be effective. The empirical section is not well reported, as the figures are mostly unreadable and poorly informative. In general, the writing of the paper is not satisfactory, with an abuse of terminology, e.g., \"hard-exploration games\" is not a common terminology, and overclaims.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "MWHgIfz4SV", "reviewer": "Reviewer_u5M2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an exploration approach based on learning multiple options, each corresponding to different intrinsic rewards. This is in contrast with the typical way of combining multiple intrinsic rewards which uses a weighted sum. The method is referred to as Deep Explore Options (DeepEO / DEO) and is an extension of Explore Options to the deep learning setting. The authors claim that DEO is more robust to ineffective or harmful intrinsic rewards, so it can be applied to a broader range of tasks without the need to tailor the set of intrinsic rewards for each task. The approach is evaluated on Atari and compared with a few baselines and ablations. ", "summary_of_the_review": "Overall, while the paper proposes a simple, intuitive approach and aims to solve an important problem, the experiments fall way short of supporting the claims of the paper. Many important baselines and ablations are missing. The paper is also quite difficult to read and the motivation isn't immediately clear. I think it also tries to make too many claims with some of them out-of-scope for the current work. I suggest focusing on a few core claims, highlighting the problem you are aiming to solve (i.e. clearly lay out the limitations of prior work and how precisely your approach addresses them), and making sure they are well-supported by the experiments. ", "strength_and_weaknesses": "# Strengths:\n1. developing a general exploration method that can work across many environments is an important problem\n2. the core idea of having separate policies that explore using different intrinsic rewards, along an exploitation policy, makes sense\n3. the proposed method is simple and intuitive\n\n# Weaknesses:\n### 1. Multiple claims are not well-supported by the experiments\n- You claim that DEO can better benefit from multiple types of exploration but never compare it with an equivalent Rainbow that uses the same number of intrinsic rewards. I suggest comparing DEO+RND+OI with Rainbow+RND+OI. \n\n- You also never demonstrate that the exploration bonuses used have complementary benefits although you mention that as a strength of DEO. Can you include DEO+OI, DEO+RND in the comparison and ideally other variants like DEO+ICM, DEM+RND+ICM, DEO+RND+ICM+OI. These also need to be compared with their Rainbow counterparts. Other exploration bonuses are fine too as long as they are orthogonal and commonly used.\n\n- You claim that DEO is more robust to harmful exploration bonuses but never compare DEO+Random with Rainbow+Random.\n\n- You claim that DEO learns transferable exploratory behaviors but you never validate this. If you want to make this claim, you need to evaluate the learned policies on new tasks and show that a policy using them learns faster than without them. \n\n- You claim that DEO extracts the exploitative behavior but this is not supported by experiments. Can you show how the agent selects the options throughout training and that towards the end it converges on always selecting the exploitative one? \n\n### 2. Limited empirical evaluation, not enough baselines or ablations\n- You only use one common intrinsic reward RND but claim that your approach can benefit from complementary intrinsic rewards, much more than standard approaches. I suggest running experiments with other common exploration bonuses such as ICM [1] or PseudoCounts \n[2], and comparing DEO with RND-only, DEO with ICM-only, DEO with both, as well as all the equivalent Rainbow \n\n- In general, policy-optimization algorithms like PPO or IMPALA tend to benefit more from intrinsic rewards than value-based algorithms like DQN or Rainbow. In fact, it is quite odd to use such exploration bonuses with DQN / Rainbow. Typically, value-based methods benefit more from things like NoisyNet [3], Bootstrapped DQN [4], ez-greedy [5] etc. Hence, I wonder if the (lack of?) benefits from DEO is mostly coming from the fact that the baselines are weak. It's possible that DEO actually improves more over the baselines when implemented on top of  a more appropriate base algorithm Thus, I recommend applying DEO to policy-optimization algorithms like PPO or IMPALA and comparing it with its weighted-sum equivalent. \n\n- You claim that $\\eta$ is vastly easier to tune and don't really consider it a HP. However, this claim isn't well supported. You need to show a sensitivity analysis with respect to $\\eta$ and $\\beta$ for the two approaches in order to evaluate whether DEO is indeed less sensitive to this HP than standard approaches are to $\\beta$. This means, showing performance of the algorithms with respect to different values of these hyperparameters for a wide enough range. \n\n### 3. Difficult to read, the motivation is not immediately clear, many terms used vaguely without definition etc.\n- please see clarity section below\n\n### 4. The empirical gains are very limited \n- from Figure 3, it looks like DEO is actually worse than Rainbow on a few hard-exploration tasks (e.g. MontezumaRevenge, Pitfall etc.) which is surprising and seems to contradict the paper's claims that DEO can better benefit from different types of exploration (which are likely to be most useful in hard-exploration environments). Can you include a more detailed discussion about this?\n- it looks like DEO is unstable on PrivateEye. Do you understand why? This is somewhat worrisome and warrants more analysis and discussion.\n\n### 5. The method is a simple extension of prior work (but this is a minor issue and not the main reason for my score)\n- I suggest outlining more clearly whether there are any substantial technical differences between DEO and EO. it's also ok if there are no major modifications, but I think it's worth making this more clear.\n\n# References\n[1] Curiosity-driven Exploration by Self-supervised Prediction, Pathak et al. 2017\n\n[2] Unifying Count-Based Exploration and Intrinsic Motivation, Bellemare et al. 2016\n\n[3] Noisy Networks for Exploration, Fortunato et al. 2017\n\n[4] Deep Exploration via Bootstrapped DQN, Osband et al. 2016\n\n[5] Temporally Extended $\\epsilon$-greedy exploration, Dabney et al. 2020\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6fUwvEIe8_g", "reviewer": "Reviewer_dS6L", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper offers a straightforward extension of the exploration options framework (Bagot et al., 2020)—which was initially introduced as a prototype for tabular environments—to a deep-learning, function approximation setting. The method is empirically evaluated in the Atari Learning Environment, following a standardized benchmark setup. The paper further considers a broader question, \"How should we incorporate intrinsic reward signals into learning algorithms?\" The authors, referring to methods that aim to answer this question as IM Incorporation methods, distill a set of normative properties that they recommend for future methods of this type. ", "summary_of_the_review": "I am recommending the rejection of this paper because it is insufficiently clear and poorly organized. Important details are often left implicit, so I often felt like I had to figure out the details via guesswork. I only found one concerning error, but because I had such difficulty understanding the work, I am not confident in being able to assess the work completely. I believe the research the authors set out to present uses fairly appropriate choices of experiments and sets out interesting recommendations for future work but the presentation itself isn't strong enough for publication at this venue.", "strength_and_weaknesses": "The exploration options framework is a very interesting setup, and this extension of the framework to a function approximation setting is an obvious next step for exploring the framework further. This is a fairly ambitious conference paper, combining both the experimental extension of exploration options to DeepEO with the more philosophical exploration of methods for incorporating intrinsic rewards into reinforcement learning agents.\n\nI considered the most prevalent weakness of this paper to be the clarity of the document. I have a number of questions coming from undefined terms and missing details.\n1. \"the types of behaviors that we can extract with IM\" (p. 1), \"DeepEOs can extract the exploiting behavior\" (p. 2) → I'm not sure what it means to extract a behaviour; can you clarify?\n2. \"However, by design they only make sense in a function approximation setting … \" (p. 2) ← I don't understand this last part of the paragraph at all.\n2. \"e.g. as a default behavior when no rewards are known.\" (p. 3) If the Exploiter's learning and decisions are based on extrinsic reward, how do you envision the Exploiter learning this kind of default behaviour? What would make it better, from the Exploiter's point of view, than choosing random actions, if \"no rewards are known\"? Or is this meant for a transfer learning setting?\n3. \"The only practical issue with such approaches that learn different agents is if the scarce relative interacting of all agents begets the necessity of Offline RL (Fujimoto et al., 2019; Levine et al., 2020)).\" (p. 3)← I don't understand why \"scarce relative interacting of all agents\" would beget \"the necessity of Offline RL.\" I'm not sure what \"relative interacting\" is, for one thing.\n4. \"This type of architecture is quite common … but very rarely so when dealing with IM, mainly because the WS is hard to scale.\" (p. 3) ← When you say \"this type of architecture,\" are you referring to architectures where the parameters of the model are shared across auxiliary tasks? This statement would be more helpful with more explanation of why scaling the WS causes particular problems in this setting beyond the usual parameter sensitivity issues associated with WS.\n5. I think the j in the loss combination function (p. 3) is supposed to be a super- or subscript—or else I'm not sure why you're multiplying by j, which is how I would read that. Also, I couldn't find definitions for your $\\mathcal L$s prior to this point. Why doesn't this equation match line 7 of Algorithm 1?\n6. I don't understand what you mean by \"environment-aware\" (p. 3)\n7. Can you provide more intuition for why the individual loss functions don't have to be balanced in the combined loss function?\n8. \"a unique buffer\" (p. 4) ← What do you mean by a unique buffer? What is it unique in comparison to?\n9. \"Instead, we note that ϵ-greedy means allocating a fraction ϵ of the agent’s time to exploring. In order to preserve this idea, we enforce our Explorer agents to occupy ϵ/2 of the agent’s time, while our Exploiter otherwise follows an ϵ/2-greedy exploration strategy.\" (p. 4) ← Can you provide more detail about how you achieved this scheduling? This detail could go in an appendix if you're worried about space requirements.\n10. What is a \"coverage map\"? (p. 4)\n11. \"Except for Solaris, the Exploiter always improves on DeepEO where option usage is forced.\" (p. 6) ← I might need your help to understand the figures this way. From my reading, there is so much variance in the performance of both Exploiter and DeepEO that it doesn't seem necessarily true that the Exploiter improves on DeepEO in any environment other than Pitfall and Freeway. What am I missing in my reading of the figures?\n12. \"we always improve on the lower bound.\" (p. 6) ← I don't understand what this means.\n\nAnd then I also have a list of parts of the paper that can be made sense of having read the whole document, but based on whether they fall in the paper, they need to be better explained. This list is not exhaustive.\n- \"relying on a single signal to help exploitation.\" (p. 1)← I found this confusing; does \"single signal\" refer to r_t? This is particularly difficult to follow being so early in the paper where you haven't introduced any alternative.\n- Please be explicit about whether \"left\" and \"right\" actions turn the agent or move the agent to a new cell, or something else?\n- Please explain the extrinsic reward function for your four-room environment\n- \"We also use it to train … an untouched DQN agent\" (p. 5) ← My best guess about what \"it\" is in this sentence is the first-cell-visit intrinsic reward, so became quite lost about the difference between the DQN+IM agent and the \"untouched DQN agent.\" \n\nOne part of the paper that I would consider an error is the treatment of the Noisy TV problem in Section 3.2. As it is described, it seems like the intention is for the third intrinsic reward function to simulate the Noisy TV problem. However, instead of setting up a reward such that one part of the environment will consistently present as a distractor generating high intrinsic reward (which is true of the \"noisy TV\"), this reward appears to be uniformly variable across each environment, which does not seem to appropriately simulate the proble.\n\nI enjoyed Section 6.2; it wasn't completely clear, but I found it a helpful overview of existing IM-Inc approaches. I also appreciated the use of the Taïga et al. (2019) benchmark to contextualize DeepEO with other IM methods.\n\nA few slight wording issues that can be read as inaccurate:\n- \"The agent is therefore deeply tied to the reward signal, and tends to fail when said signal is sparse or noisy.\" (p. 1) ← I suggest rephrasing; while this type of failure is indeed typical of present RL agents, this phrasing makes it sound like the problem is inherent to the RL framework.\n- \"IM biologically refers to the natural tendency of organisms to explore.\"  (p. 1) ← This characterism isn't necessarily accurate. One of the classical definitions of intrinsic motivation is 'engagement in an activity for its own sake,' (Deci, 1975, p. 23) which doesn't perfectly align with the way the word 'explore' is typically used, especially in the context of reinforcement learning.\n- \"i.e. additional actions,\" (p. 1) ← It is a bit misleading to simply call options additional actions, as they have a very different structure from standard actions (I understand that they are chosen by the Exploiter just like low-level actions) so I'd recommend rephrasing.\n- \"to explore for a fixed amount of time.\" (p. 1) ← You may want to avoid making it sound like the 'fixed time' part is inherent to your definition of exploration options; while the prototype implementation of exploration options provided by Bagot et al. (2020, p. 4) does indeed only use them to explore for a fixed amount of time, they imply that the theoretical architecture could support more general termination conditions.\n- \"Motivating exploration is by far the most common goal then, and is divided into…\" (p. 7) ← While I agree that this is the division that Aubret et al. (2019) make, the phrasing makes it sound like it is the only possible division, when there are other works that have used other distinctions to taxonomize intrinsic rewards (e.g. Oudeyer & Kaplan, 2009)\n\nTypos and grammatical suggestions (no intended influence on score)\n- \"to the a weighted\" (p. 1)→ \"to a weighted\"\n- The replay buffer in Figure 1 could probably use some more ellipses, ideally, like \"s, a, r^e, r^{i1}, r^{i2}, … , s' \" \n- \"the Explore Option\" (p. 2) → might make more sense as plural; aren't there multiple Explore Options?\n- \"the IM rewards themselves guiding\" (p. 3) → \"the IM rewards guiding\"\n- \"3 majors improvements\" (p. 4) → \"3 major improvements\"\n- \"less-visited states, that\" (p. 4) → \"less-visited states that\" or \"less-visited states, which\"\n- \"explicited\" (p. 4) → explicit is an adjective, rather than a verb; maybe you want \"shown\" or something similar\n- \"we evaluate our Exploiter\" (p. 5) → \"we also evaluate our Exploiter\" (to help make it clear that you are also evaluating DeepEO while retaining the options—in fact, adding an explicit sentence noting that you are including DeepEO in the evaluations wouldn't go amiss!)\n- \"Guillaume Lample and Devendra Singh Chaplot. Playing fps games …\" (p. 11) → \"FPS games\"\n- \"and lead to\" (p. 14) → \"and led to\"\n\n\nDeci, E.L. (1975). Conceptualizations of Intrinsic Motivation. In: Intrinsic Motivation. Springer, Boston, MA. doi:10.1007/978-1-4613-4446-9_2", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4fwHRbJiPz", "reviewer": "Reviewer_xbC8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper extends the idea of ExploreOptions to the deep RL setting. Four different sub-policies are trained off-policy on the same data: an Exploit policy, and 3 Explore policies, which each use a different intrinsic motivation (IM) method to incentivize exploration (e.g. including RND). Results are presented in a gridworld and Atari.", "summary_of_the_review": "Overall, while the idea behind the paper is compelling, it has been explored in prior work and was not presented in a compelling way here. The results indicate the proposed technique is less effective than well-known existing methods, limiting the potential impact of the paper. \n\nIf the authors can clarify some points I may be willing to increase my score. Can you please address:\n- How this work is different from Campos et al. 2021 and what additional contributions does it make?\n- Is this a hierarchical method in which a central controller learns which option to invoke, or does the exploit option learn to invoke the others (in which case Figure 2 should be revised), or are the options randomly invoked? \n- Why focus on three similar exploration IM methods to create the options and not include empowerment? ", "strength_and_weaknesses": "A strength of the paper is that the idea is compelling. Learning both exploit and explore options and being able to intelligently switch between them is a more principled approach to the exploration/exploitation problem than combining incentives to simultaneously explore and exploit into one reward. \n\nThere are three main weaknesses of the paper:\n1) Novelty: the central idea of the paper appears to have been explored previously in Explore Options (Bagot et al. 2020) (albeit not with deep learning), and Campos et al. (2021). Particularly with Campos et al., it appears to also use exploratory options which are called by an exploiter agent, so the contribution of the current paper over Campos is not clear. This should be more thoroughly explained. \n\n2) Effectiveness/Impact: The results on Atari reveal that DeepEO does not work effectively compared to the Rainbow or Rainbox+RND baselines. It appears that DeepEO provides a significant performance improvement above the baselines in only 2/11 environments, and these are both \"easy\" exploration tasks as described by the authors. These results are not compelling. \n\n3) Clarity: The actual method used for DeepEO is unclear, as I will explain below. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject