When should agents explore?
Authors: Miruna Pislar
David Szepesvari
Georg Ostrovski
Diana L Borsa
Tom Schaul
Venue: ICLR 2022 Spotlight (2022)
Keywords: exploration
mode-switching
reinforcement learning
Atari

Abstract: Exploration remains a central challenge for reinforcement learning (RL). Virtually all existing methods share the feature of a *monolithic* behaviour policy that changes only gradually (at best). In contrast, the exploratory behaviours of animals and humans exhibit a rich diversity, namely including forms of *switching* between modes. This paper presents an initial study of mode-switching, non-monolithic exploration for RL. We investigate different modes to switch between, at what timescales it makes sense to switch, and what signals make for good switching triggers. We also propose practical algorithmic components that make the switching mechanism adaptive and robust, which enables flexibility without an accompanying hyper-parameter-tuning burden. Finally, we report a promising initial study on Atari, using two-mode exploration and switching at sub-episodic time-scales.

Published as a conference paper at ICLR 2022
WHEN SHOULD AGENTS EXPLORE?
Miruna Pislar
David Szepesvari
Georg Ostrovski
Diana Borsa
Tom Schaul
DeepMind London, UK fmirunapislar, dsz, ostrovski, borsa, (eem))ddeemiid.comm com
ABSTRACT
Exploration remains central challenge for reinforcement learning (RL), Virtu- ally all existing methods share the feature a monolithic behaviour policy that changes only gradually (at best). In contrast, the exploratory behaviours of an- imals and humans exhibit a rich diversity, namely including forms of switching between modes. This paper presents monolithic exploration for RL. We investigate different modes to switch between, at what timescales it makes sense to switch, and what signals make for good switching triggers. We also propose practical algorithmic components that make the switching mechanism adaptive and robust, which enables flexibility without an accompanying hyper-parameter parameter-tuning burden. Finally, we report a promis- ing and detailed analysis on Atari, using woomoode exploration and switching sub-episodic time-scales.
initial study of mode-switching, non-
INTRODUCTION
The trade-off between exploration and exploitation is described as the crux of learning and behaviour across many domains, not just reinforcement learning (Sutton & Barto 2018), but also in decision making (Cohen et al. 2007), evolutionary biology (Cremer al.] 2019), ), ecology (Kembro al., 2019). neuroscience (e.g., focused versus diffuse search in visual attention (Wolfe et al.) 1989). dopamine regulations (Chakroun et al., (2200). cognitive sciences (Hills et al.J [015),
well as
psychology and psychiatry (Addicott et al.) (017). In nutshell, exploration is about the balance between taking the familiar choice that is known to be rewarding and learning about unfamiliar options of uncertain reward, but which could ultimately be more valuable than the familiar options.
Ample literature has studied the question of how much to explore, that is how to set the overall trade-off (and how to adjust it over the course of learning) (Jaksch et al. 2010; Cappé et al.) 2013 Lattimore Szepesvári 2020 Thrun 1992), and the question of how to explore, namely how to choose exploratory actions (c.g., randomly, optimistically. intrinsically motivated, otherwise) (Schmidhuber 1991 Oudeyer& & Kaplan Linke et al., 2019). In contrast, the question of when explore has been studied very little, possibly because it does not arise in bandit problems, where a lot of exploration methods are rooted. The 'when' question and its multiple facets are the subjects of this paper. We believe that addressing it could lead more intentional forms of exploration.
Consider an agent that has access to two modes of behaviour, an 'explore' mode and an 'exploit' mode (e.g., a random policy and greedy policy, as in e-greedy). Even when assuming that the overall proportion of exploratory steps is fixed, the agent still has multiple degrees of freedom: it can explore more at the beginning of training and less later phases; it may take single exploratory steps or execute prolonged periods of exploration; it may prefer exploratory steps early or late within an episode; and it could trigger the onset (or end) of an exploratory period based on various criteria. Animals and humans exhibit non-trivial behaviour in all of these dimensions, presumably encoding useful inductive biases that way (Power) 1999). Humans make use of multiple effective strategies, such as selectively exploring options with high uncertainty (a form of directed, information- seeking exploration). and increasing the randomness their choices when they are more uncertain (Gershman 2018 Gershman & Tzovaras 2018 Ebitz al. 2019). Monkeys use directed explo- ration to manage explore-explos trade-offs, and these signals are coded in motivational brain regions (Costa et al. 2019). Patients with schizophrenia register changes in directed exploration and expe-

---

Published as a conference paper at ICLR 2022
rience low-grade inflammation when shifting from exploitation to random exploration (Waltz al. 2020 Cathomas et al.] 2021). This diversity is what motivates us to study which of these can benefit RL agents in turn, by expanding the class of exploratory behaviours beyond the commonly used monolithic ones (where modes are merged homogeneously in time).
2
METHODS
The objective of an RL. agent is to learn policy that maximises external reward. At the high level, it achieves this by interleaving two processes: generating new experience by interacting with the environment using a behaviour policy (exploration) and updating its policy using this experience (learning). As RL is applied to increasingly ambitious tasks, the challenge for exploration becomes to keep producing diverse experience, because something has not been encountered, it cannot be learned. Our central argument is therefore simple: monolithic, time- hime-homogeneous behaviour policy is strictly less diverse than a heterogeneous mode-switching one, and the former may ham- string the agent's performance. As illustrative example, consider human learning how to ride bike (explore), while maintaining their usual happiness through food, sleep, work (exploit): there is stark contrast between monolithic, handlebar or a turn of a pedal once every few minutes or so, and the mode-switching behaviour that dedicates prolonged periods of time exclusively to acquiring the new skill of cycling.
homogenogs behaviour that interleaves a twist of the
While the choice of behaviour in pure exploit mode is straightforward, namely the greedy pursuit of external reward (or best guess thereof), denoted by G. there are numerous viable choices for behaviour in pure explore mode (denoted by X'). In this paper we consider two standard ones: Xv. the naive uniform random policy, and X,, an intrinsically motivated behaviour that exclusively pursues a novelty measure based on random network distillation (RND, (Burda et al., 2018)). See Section and Appendix B for additional possibilities of X. In this paper we choose fixed behaviours for these modes, and focus solely the question of when to switch between them. In our setting, overall proportion of exploratory steps (the how much), denoted by px, is not directly controlled but derives from the when.
2.1 GRANULARITY
An exploration period is an uninterrupted sequence of steps explore mode. We consider four choices of temporal granularity for exploratory periods, also illustrated in Figure
Step-level exploration the most common scenario, where the decision to explore is taken inde- pendently at each step. affecting one action] The canonical example is E-greedy [Fig[C)
Experiment-level exploration is the other extreme, where behaviour during training produced in explore mode, and learning is off- oof-policy (the greedy policy is only used for evaluation). This scenario also very common, with most forms of intrinsic motivation falling into this category, namely pursuing reward with intrinsic bonus throughout training (Fig A
Episode-level exploration is the case where the mode fixed for an entire episode at time (e.g., training games versus tournament matches in sport), see Fig. This has been investigated for simple cases, where the policy's 's level of stochasticity is sampled at the beginning of cach episode (Horgan et al., 2018 Kapturowski et al. 2019 Zha et al. 2021).
Intra-episodic exploration is what falls in-between step- and episode-level exploration, where ex- ploration periods last for multiple steps, but less than full episode. This is the least commonly studied scenario, and will form the bulk of our investigations D.E.F.G.
We denote the length of exploratory period by n. (and similarly ng for exploit mode). To characterise granularity, our summary statistic of choice is medx == median(nx). Note that there two possible units for these statistics: the raw steps or the proportion of the episode length L.
1 'The length of exploratory period tends to he short, but it be greater than multiple consecutive step-wise decisions to explore can create longer periods. NNtt that it is also possible to interpret E-greedy as experiment-level exploration, where the X' policy is fixed to noisy version of G.
2

---

Published as a conference paper at ICLR 2022
episode boundaries
explore mode X'
exploit mode g
A
B
C
PX
D
E
DE
F
stap
med
episacts
G
Figure 1: Illustration different types of temporal structure for two-mode exploration. Left: Each line A-G depicts an excerpt of an experiment (black lines show episode boundaries, experiment continues on the right), with colour denoting the active mode (blue exploit, magenta is explore). A is of experiment-level granularity, B episode-level, C step-level, and D-G are of intra-episodic exploration granularity. Right: The same examples, mapped onto characteristic plot of summary statistics: overall exploratory proportion px versus typical length of an exploratory period medx. The yellow-shaded area highlights the intra-episodic part of space studied in this paper (some points not realisable, e.g., when px then med must be large). C, D, E, F share the same Px = 0.2, while interleaving exploration modes in different ways. D and E share the same modx value, and differ only whether exploration periods spread out, happen toward the end.
The latter has different (relative) semantics, but may more appropriate when episode lengths vary widely across training. We denote it as rmedx == median(nxx/L).
2.2 SWITCHING MECHANISMS
Granularity is but the coarsest facet of the 'when' question, but the more precise intra-cpisode tim- ings matter too, namely when exactly to start and when stop an exploratory period. This section introduces two mechanisms, blind and informed switching. It is worth highlighting that, in general, the mechanism (or its time resolution) for entering explore mode differs from the one for exiting (to enter exploit mode) - this asymmetry is crucial to obtain flexible overall amounts of exploration. switching were symmetric, the proportion would be px 0.5.
Blind switching The simplest switching mechanism does not take any state into account (thus, we call it blind) and is only concerned with producing switches at some desired time resolution. It can be implemented deterministically through counter (c.g., enter explore mode after 100 exploit mode steps), or probabilistically (e.g., at each step, enter explore mode with probability 0.01). Its expected duration be parameterised in terms of raw steps, in terms of fractional episode length. The opposite blind switching is informed switching, as discussed below.
Informed switching Going beyond blind switching opens up another rich set of design choices, with switching informed by the agent's internal state. There are two parts: first, a scalar trigger signal is produced by the agent each step, using its current information drawing inspiration from human behaviour, we view the triggering signal as a proxy for uncertainty (Schulz et al. 2019): when uncertainty is high, the agent will switch to explore. Second, a binary switching decision taken based on the trigger signal (for example, by comparing it to a threshold). Again, triggering will generally not symmetric between entering and exiting an exploratory period.
To keep this paper focused, we will look at one such informed trigger, dubbed "value promise dis- crepancy' (see Appendix B for additional competitive variants). This is an online proxy how much of the reward that the agent's 's past value estimate promised (k steps ago) have actually come about. The intuition is that uncertain parts of state space, this diserepancy will generally be larger than when everything goes as expected. Formally,
-
D Prromise k,t) = V(s
1=0
where V(s) is the agent's value estimate state 8, R is the reward, and y is discount factor.
3

---

Published as a conference paper at ICLR 2022
Starting mode When periods last for significant fraction of episode length, it also matters how the sequence is initialised, ie., whether an episode starts in explore or in exploit mode, or more generally, whether the agent explores more early in an episode or more later on. It is conceivable that the best choice among these is domain dependent (see Figure 6 in most scenarios, the states the beginning of an episode have been visited many times, thus starting with exploit mode can be beneficial; in other domains however, early actions may disproportionately determine the available future paths (e.g., build orders in StarCraft (Churchill & Buro 2011
2.3 FLEXIBILITY WITHOUT ADDED BURDEN
Our approach introduces additional flexibility to the exploration process, even when holding the specifics of the learning algorithm and the exploration mode fixed. To prevent this from turning into an undue hyper-parameter tuning burden, we recommend adding two additional mechanisms.
Bandit adaptation The two main added degrees of freedom in our intra-episodic switching set-up are when (or how often) to enter explore mode, and when (or how quickly) to exit it. These can be parameterised by a duration, termination probability or target rate (see Section 3.T). In either case, we propose to follow [Schaul et al.] (2019) and [Badia et al. (2020a), and delegate the adaptation of these settings to meta-controller, which is implemented as a non-stationary multi-armed bandit that maximises episodic return. As added benefit, the 'when' of exploration can now become adaptive to both the task, and the stage of learning.
Homeostasis In practice, the scales of the informed trigger signals may vary substantially across domains and across training time. For example, the magnitude of D promise will depend on reward scales and density and can decrease over time accuracy improves (the signals could also be noisy). This means that naively setting a threshold hyper-parameter is impractical. For a simple remedy, we have taken inspiration from neuroscience (Turrigiano & Nelson (004) to add homeostasis to the binary switching mechanism, which tracks recent values of the signal and adapts the threshold for switching so that a specific average target rate is obtained. This functions as an adaptive threshold, making tuning straightforward because the target rate of switching is configured independently of the scales of the trigger signal. See Appendix A for the details of the implementation.
3 RESULTS
The design space we propose contains a number of atypical ideas for how to structure exploration. For this reason, we opted to keep the rest of our experimental setup very conventional, and include multiple comparable baselines, ablations and variations.
subset of games of the Atari Learn- ing Environment (Bellemare et al. 2013), a common benchmark for the study of exploration. All experiments are conducted across games (FROSTBITE, GRAVITAR, H.E.R.O., MONTEZUMA'S REVENGE, Ms. PAC-MAN, PHOENIX, STAR GUNNER), the first of which are classified as hard exploration games (Bellemare et al. 2016), using 3 seeds per game. For our agent, use the R2D2 architecture (Kapturowski et al. 2019), which is a modern, distributed version of DQN (Mnih et al.) 2015) that employs recurrent network approximate its Q-value function. This is common basis used in exploration studies (Dabney et al. 2020; Badia et al. 20200b a). The only major modification conventional R2D2 is its exploration mechanism, where instead we implement all the variants of mode-switching introduced in Section 2 Separately from the experience collected for learning, we run an evaluator process that assesses the performance of the current greedy policy. This is what we report in all our performance curves (see Appendix A for more details).
Setup: R2D2 on Atari We conduct our investigations
There are a few simple baselines worth comparing to, namely the pure explore mode (px = 1, Fig T A) and the pure exploit mode (pxx 0), as well as the step-wise interleaved e-greedy execution (Fig CC) where p. = 0.01 = S (without additional episodic or intra-episodic structure). Given its wide adoption in well-tuned prior work, expect the latter to perform well overall. The fourth baseline picks a mode for an entire episode at a time [Fig[B). with the probability of pick- ing X' being adapted by bandit meta-controller. We denote these experiment-levee exper r simeitttt-- - leeeeeeeeee G-G. step-level- 01 and poisode-levell respectively. For each of these, we have version with uniform (Xy) and intrinsic (Xx) explore mode.
Baselines
4

---

Published as a conference paper at ICLR 2022
explore mode Xy
first mode greedy
XU-intra (100, bl ind, p
exploration periods end after 100 steps
trigger exploration blindly according to bandit (*) over probabilities (p)
Figure 2: Illustrating the space of design decisions for intra-episodic exploration (see also Figure
3.1 VARIANTS OF INTRA-EPISODIC EXPLORATION
As discussed Section 2 there are multiple dimensions along which two-mode intra- ex- ploration can vary. The concrete ones for experiments are:
Explore mode: uniform random Ny, RND intrinsic reward X2 (denoted XU and XI).
of these is adaptively picked by bandit (denoted by *), or the switching is symmetric between entering and exiting explore mode (denoted by =).
Explore duration (nx): this can be fixed number of steps (1, 10, 100),
Trigger type: either blins blind or informed (based on value promise, see Section 2.2
Exploit duration (ng): for blind triggers, the exploit duration can be parameterised by fixed number of steps (10, 100, 1000, 10000), indirectly defined by probability of terminating (0.1, 0.01, 0.001,0.0000 0.0001), or adaptively picked by bandit over these choices (denoted by n or p*, respectively). For informed triggers, the exploit duration is indirectly parame- terised by a target rate in (0.1, 0.01, 0.001, 0.0001), or a bandit over them (p*), which is in turn transformed into adaptive switching threshold by homeostasis (Section 2.2
Starting mode: 9 greedy (default) or X' explore (denoted G or X).
We can concisely refer to particular instance by tuple that lists these choices. For example, XU- ra (100, in rmed, p*, x) denotes uniform random exploration Xys, with fixed 100- step explore periods, triggered by the malue-promise signal at bandit-detrmmined rate, and starting in explore mode. See Figure 2 for an illustration.
3.2 PERFORMANCE RESULTS
We start by reporting overall performance results, to reassure the reader that our method viable (and convince them keep reading the detailed and qualitative results in the following sec- tions). FigureßJshww performance across 7 Atari games according to two human-normalised aggre- gation metries (mean and median), comparing one form of intra-episodic exploration to all the base- lines, separately for each explore mode (Xy and X1). The headline result is that intra-episodic ex- ploration improves over both step-level and episode-level baselines (as well the pure experiment- level modes that we would not expect to be very competitive). The full learning curves per game are found in the appendix, and show scores on hard exploration games like MONTEZUMA'S REVENGE or PHOENIX that are also competitive in absolute terms (at our compute budget of 2B frames).
Note that there is subtle difference the learning setups between Xy and X,, as the latter requires training separate head to estimate intrinsic reward values. This is present even in pure exploit mode, where it acts as an auxiliary task only (Jaderberg et al. 2016), hence the differences pure greedy curves in Figure 3 For details, Appendix
3.3 DIVERSITY RESULTS
In study like ours, the emphasis is not measuring performance, but rather characterising the diversity of behaviours arising from the spectrum of proposed variants. A starting point is to return Figure [ (right), and assess how much of the previously untouched space is now filled
5

---

Published as a conference paper at ICLR 2022
Uniform random esploration
Intrinsic reward exploration
Modian
Modion
Mistes d
122
30 esseed loves
122
103
Figure 3: Human-normalized performance results aggregated over Atari games and seeds, com- paring the four levels of exploration granularity. Left two: uniform explore mode Nv. Right two: RND intrinsic reward explore mode X/. In each case, the baselines pure modes X and G, step- level switching with e-greedy, and episodic switching (with bandit-adapted proportion). Note that the X2 t7--ste--leee experiment uses both an intrinsic and an extrinsic reward, as Burda 2018
frestahte
30-2
0.2
0.4
0.6
0.0
0.2
0.4
0.8
1.0
0.2
0.4
1.0
0.4
9.6 meedi
0.5
medd
mmeeS
medi
2030
Wilemat ritmmad 301.2)
10000
temee on informad.t.. an 1are
Innne
2000
1000
56006
Tredifate
morkerums
1011
10-2
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.5
1.0
0.5
0.2
0.4
0.5
1.0
0.2
0.4
0.6
0.8
shoemb
123m
$330
4000
Informeundt 1 Callnci I2
300
X0C
Figure 4: Rows and 3: Summary characteristics Px and rmedx of induced exploration behaviour, for different variants of intra-episodic exploration (and episodic baseline for comparison), subset of Atari games. Bandit adaptation can change these statistics over time, hence square and cross markers show averages over first and last 10% of training, respectively. Rows 2 and 4: Corresponding final scores (averaged over final 10% of training). Error bars show the span between min and max performance across seeds. Note how different variants cover different parts of characteristic space, and how the bandit adaptation shifts the statistics into different directions for different games. See main text for further discussion and Appendix C for other games and variants.
by intra-episodic variants, and how the 'when' characteristics translate into performance. Figure 4 answers these questions, and raises some new ones. First off, the raw amount of exploration px is not sufficient predictor of performance, implying that the temporal structure matters. It also shows substantial bandit adaptation at work: compare the exploration statistics at the start (squares) and end and-points of training (crosses), and how these trajectories differ per game; common pattern that reducing px far below 0.3 is needed for high performance. Interestingly, these adaptations similar between X and X,, despite very different explore modes (and differing performance

---

Published as a conference paper at ICLR 2022
XU-intra(10 wttpe11..iiiido pibiidddn...0 level=frostbite
XU-intra(10 intornodddppp.... level=frostbite
Frostbite
17500
15000
Indiv apicodos
12300
Egisode
7500
XU-intra(10 blind,p .G)
XU-intra(10, informeedd
200
500
1000
1200
1400
750
1500
Episode Steps
Episode Steps
Learner steps
X blind,ppr p)
XU-intra(10 informee, p*.G)
leveligravitar
level gravitar
Gravitar
oodiual apicdde
4000
Episode
3000
2000
1000
XU-intra(10, (iiittra(0,blinddp..... blidd,iid,p.i innfotredd.iffommmed..o informed.dp....... .*.))
0
200
400
800
800
200
400
600
100
1000
0.0
0.5
1.0
15
Episode Steps
Episode Stego
Learner steps
365
Figure 5: Left and center: Illustration of detailed temporal structure within individual episodes, on FROSTBITE (top) and GRAVITAR (bottom), contrasting two trigger mechanisms. Fach subplot shows 15 randomly selected episodes (one per row) that share the same overall exploration amount px 0.1. Each vertical bar (magenta) represents exploration period of fixed length nx 10; each blue chunk represents an exploitation period. Left: blind, step-based trigger leads to equally spaced exploration periods. Center: trigger signal informed by value promise leads to very different within-episode patterns, with some parts being densely explored, and others remaining in exploit mode for very long. Right: the corresponding learning curves show a clear performance benefit for the informed trigger variant (orange) in this particular setting. Appendix C has similar plots for many more variants and games.
frostbite
frostbite
0.
star_gurber
gravitar
star_gna star_gunter
0.6
gravitar
htt.iirrrappllll.. www.lppppddddd... Ad--intrafp.blinde X)
intraa(pbbindd,pt blind,pt.C XI-intra(p, ppiddd, X)
phoe
phoen
heoo
ms. pas_paccan.__ montezuma_revene
ms. coss_pacina.... mwnteeuima revenge
Figure 6: Starting mode effect. Final mean episode return for two blind intra-episode experiments that differ only in start mode, greedy (blue) or explore (orange). Scores are normalised so that is the maximum result across the start modes. Either choice reliably boost harm performance, depending on the game. Left: uniform explore mode Xy. Right: intrinsic reward explore mode X1.

---

Published as a conference paper at ICLR 2022
$1000
Frestotte
9332
8000
68
7002
6000
0930
Episods
4000
8000
64
4220
2000
3000
2032
XUinntral 400.dd0001... prob-based
XU-intraa- www.ddi........... stop based
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.5
3.0
125
PX
PX
Contrasting the behavioural characteristics between two forms of blind switching, step-based (left) and probabilistic (center), on the example of FROSTBITE. Each point is an actor episode, with colour indicating time in training (blue for early. red for late). Note the higher diversity of px when switching probabilistically. Right: Corresponding performance curves indicate that the probabilistic switching (red) has a performance benefit, possibly because it creates the opportunity for 'lucky' episodes with much less randomness in a game where random actions can casily kill the agent. For more games, please see the Appendix C
Figure 7: Left and center:
results). We would expect prolonged intrinsic exploration periods to be more useful than prolonged random ones, and indeed, comparing the high-rmedy variant (purple) across Xy and X', it appears more beneficial for the latter. Zooming in on specific games, a few results stand out: in Ny mode, the only variant that escapes the inherent local optimum of PHOENIX is the blind, doubly adaptive one (purple), with the bandits radically shifting the exploration statistics over the course of training. In contrast, the best results on MONTEZUMA'S REVENGE are produced by the symmetric, informed trigger variant (light green), which is forced to retain a high Px- Finally, FROSTBITE is the one game where an informed trigger (red) clearly outperforms its blind equivalent (purple).
These insights are still limited to summary statistics, so Figure looks in more depth at the detailed temporal structure within episodes (as in Figure [ left). Here the main comparison is between blind and informed triggers, illustrating that the characteristics of the fine-graine ane-ggri within-episode structure can differ massively, despite attaining the same high-level statistics px and mcdx. We can see quite lot of variation in the trigger structure the moments we enter exploration are not evenly spaced anymore. As a bonus, the less rigid structure of the informed trigger (and possibly the more carefully chosen switch points) end up producing better performance too.
Figure 6 sheds light on a complementary dimension, differentiating the effeets of starting in explore or exploit mode. In brief, each of these be consistently beneficial in some games, and consis- tently harmful in others. Another observation here is the dynamics of the bandit adaptation: when starting in exploit mode, it exhibits preference for long initial exploit periods in many games (up to 10000 steps), but that effect vanishes when starting in explore mode (see also Appendix C.. More subtle effects arise from the choice of parameterisation of switching rates. Figure 7 shows stark qualitative difference on how probabilistic switching differs from step-count based switching, with the former spanning a much wider diversity of outcomes, which improves performance.
3.4 TAKE-AWAYS
Summarising the empirical results in this section, two messages stand out. First, there seems to be a sweet spot in terms of temporal granularity, and intra-episodic exploration the right step towards finding it Second, the vastly increased design space of our proposed family of methods gives rise to a large diversity of behavioural characteristics; and this diversity not superficial, also translates to meaningful performance differences, with different effects in different games, which cannot be reduced to simplistic metrics, such as px. In addition, we provide some sensible rules-of-thumb for practitioners willing to join to let bandit figure out the precise settings, but it is worth curating its choices to at most a handful. Jointly using two bandits across factored dimensions is very adaptive, but can sometimes be harmful when they decrease the signal- comal-to-noon noise ratio in each other's 's learning signal. Finally, the choice of the uncertainty- based trigger should be informed by the switching modes (see Appendix B for details).
the journey of intra-episodic exploration. In general, it is useful
8

---

Published as a conference paper at ICLR 2022
4 DISCUSSION
Time-based exploration control The emphasis of our paper was on the benefits of heterogeneous temporal structure in mode-switching exploration. Another potential advantage over monolithic approaches is that it may be casier to tune trper-parameters related to an explicit exploration budget (e.g., via px) than tune intrinsic reward coefficient, especially if extrinsic reward scales change across tasks or time, and if the non-stationarity of the intrinsic reward affects its overall scale.
Diversity for diversity's 's sake One role of ceneral-pupooss exploration method is to allow an agent to get off the ground in a wide variety of domains. While this may clash with sample-efficient learning on specific domains, we believe that the former objective will come to dominate in the long run. In this light, methods that exhibit more diverse behaviour are preferable for that reason alone because they are more likely to escape local optima or misaligned priors.
Related work While not the most common approach to exploration in RL, we are aware of some notable work investigating non-trivial temporal structure. The ez-greedy algorithm (Dabney et al. 2020) initiates contiguous chunks of directed behaviour ('flights') with the length sampled from a heavy-tailed distribution. In contrast to our proposal, these flights act with a single constant action instead of invoking an explore mode. Campos et al. (2021) pursue a similar idea, but with flights along pre-trained coverage policies, while Fcoffet et al. (202T) chain 'return-to-state policy to an explore mode. Maybe closest to our X3 setting is the work of Bagot et al. (2020), where periods of intrinsic reward pursuit are explicitly invoked by the agent. Exploration with gradual change instead of abrupt mode switches generally appears at long time-scales, such when pursuing intrinsic rewards (Schmidhuber) 2010 Oudeyer & Kaplan 2009) but can also be effective at shorter time- scales, e.g., -iver--GveeUU (Badia et al.] (2020b). Related work on the question of which states to prefer for exploration decisions (Tokic) 2010) tends not to consider prolonged exploratory periods.
Relation to options Ideas related to switching behaviours at intra-episodic time scales are well- known outside of the context of exploration. In the options framework in hierarchical RI., the goal is to chain together sequence of sub-behaviours into hewarrdmaximising policy (Sutton al.) 1999 Mankowitz et al., 2016). Some work has looked at using options for exploration too (Jinnai et al., 2019a] Bougie Ichise 202T). In its full generality, the options framework a substantially more ambitious endeavour than our proposal, as it requires learning full state-dependent hierarchical policy that picks which option to start (and when), as well as jointly learning the options themselves.
Limitations Our proposed approach inherits many typical challenges for exploration methods, such as sample efficiency or trading off risk. An aspect that particular to the intra-episode switch- ing case is the different nature of the off- ffff-poiiy-ness... The resulting effective policy can produce state distributions that differ substantially from those of either of the two base mode behaviours that are being interleaved. It can potentially visit parts of the state space that neither base policy would reach if followed from the beginning of the episode. While a boon for exploration, this might pose a challenge to learning, it could require off ofl-policy corrections that treat those states differently and do not only correct for differences in action space. Our paper does not use (non-trivial) off-policy correction (see Appendix ( our initial experimentation showed it is not essential component in the current setting (see Figure 15) We leave this intriguing finding for future investigation.
Future work With the dimensions laid out in Section 2 should be clear that this paper can but scratch the surface. We see numerous opportunities for future work, some of which we already carried out initial investigations (see Appendix [B). For starters, the mechanism could go beyond two-mode and switch between exploit, explore, novelty and mastery (Thomaz& & Breazeal 2008
or between many diverse forms of exploration, such as levels of optimism (Derman et al. 2020
Moskovitz et al. (021). Triggers could be expanded or refined by using different estimations of uncertainty, such as ensemble discrepancy (Wiering Van Hasselt 2008 [Buckman et al.) 2018),
amortised value errors (Flennerhag et al. 20200) density models (Bellemare al. [216]] Ostrovski ct al.. 2017); or other signals, such as salience (Downar et al. 20022, , minimal coverage (Jinnai et al. 2019a b). or empowerment (Klyubin et al. 2005| Gregor al. 2016 Houthooft et al. 2016).
Conclusion We have presented an initial study of intra-episodic exploration, centred on the sce- nario of switching between an explore and an exploit mode. We hope this has broadened the avail- able forms of temporal structure in behaviour, leading to more diverse, adaptive and intentional forms of exploration, in turn enabling RL. to scale ever more complex domains.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We would like to thank Audrunas Gruslys, Simon Osindero, Eszter Vértes, David Silver, Dan Hor- gan, Zita Marinho, Katrina McKimney, Claudia Clopath, David Amos, Víctor Campos, Remi Munos, and the entire DeepMind team for discussions and support, and especially Pablo Sprechmann and Luisa Zintgraf for detailed feedback on an earlier version.
REFERENCES
M. A. Addicott, J. M. Pearson, M. M. Sweitzer, D. L. Barack, and M. Platt. A Primer Foraging and the Explore/Exploit Trade-Off for Psychiatry Research. Neuropychopharmacooeg, 42(10): 1931-1939, Sep 2017.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the Atari human benchmark, 2020a.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies, 2020b.
Louis Bagot, Kevin Mets, and Steven Latré. Learning intrinsically motivated options to stimulate policy exploration, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Marc G Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Neural Information Pro- cessing Systems, 2016.
Nicolas Bougie and Ryutaro Ichise. Fast and slow curiosity for high-level exploration reinforce- ment learning. Applied Intelligence, 51(2):1086-1100 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: 7/ //qithub. cowppoogl///oo/....o...o
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. arXiv preprint a7Xiv:1807.0167. 2018.
David Budden, Matteo Hessel, lurüi Kemaev, Stephen Spencer, and Fabio Viola. Chex: Testing made fun, in jax!, 2020a. URL URL/bttt://iithbb.comm http://c///deeeiidd/eeemm hearepeiiind/exxxx
David Budden, Matteo Hessel, John Quan, Steven Kapturowski, Kate Baumli, Surya Bhupatiraju, Aurelia Guy, and Michael King. RLax: Reinforcement Learning in JAX, 2020b. URL http:
//github.com dee http://ddemmiid///ddd/dm
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. CoRR, abs/1810.1 18944, 2018.
Víctor Campos, Pablo Sprechmann, Steven Hansen, Andre Barreto, Steven Kapturowski, Alex Vitvitskyi, Adrià Puigdomènech Badia, and Charles Blundell. Coverage as principle for discov- ering transferable behavior in reinforcement learning. arXiv preprint 13355:2102.13515, 2021.
Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, Gilles Stoltz, et al. upper confidence bounds for optimal sequential allocation. Annals of Statis- fics, 11541111511611 2013.
10

---

Published as a conference paper at ICLR 2022
Albin Cassirer, Gabriel Barth-Maron, Thibault Sottiaux, Manuel Kroiss, and Eugene Brevdo. Re-
verb: An efficient data storage and transport system for ml research, 2020. URL https:
//github. com /
Flurin Cathomas, Federica Klaus, Karoline Guetter, Hui-Kuan Chung, Anjali Raja Beharelle, To- bias R. Spiller, Rebecca Schlegel, Erich Seifritz, Matthias N. Hartmann-Riemed Philippe N. To- bler, and Stefan Kaiser. Increased random exploration in schizophrenia is associated with inflam- mation. npj Schizophrenia, 7(1):6, Feb 2021.
K. Chakroun, D. Mathar, A. Wiehler, F. Ganzer, and Peters. Dopaminergic modulation of the exploration/exploittion trade-off in human decision decision-making Elife, 9, 06 2020.
David Churchill and Michael Buro. Build order optimization in starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 6(1), 2011.
Jonathan D Cohen, Samuel M McClure, and Angela J Yu. Should stay or should go? How the human brain manages the trade-off between exploitation and exploration. Philosophical Trans- actions of the Royal Society B: Biological Sciences, 362 9143999993-92,,, 2007.
Vincent D. Costa, Andrew R. Mitz, and Bruno B. Averbeck. Subcortical substrates of explore-exploit decisions primates. Neuron, 103(3):533-555.) August 2019.
Jonas Cremer, Tomoya Honda, Ying Tang, Jerome Wong-Ng, Massimo Vergassola, and Terence Hwa. Chemotaxis as navigation strategy boost range expansion. Nature, 575(7784):6588 663, 11 2019.
Will Dabney, Georg Ostrovski, and André Barreto. Temporally-extended c-greedy exploration, 2020.
Esther Derman, Daniel Mankowitz, Timothy Mann, and Shie Mannor. A bayesian approach robust reinforcement learning. Uncertainty in Artificial Intelligence, pp. 648-658. PMLR, 2020.
Jonathan Downar, Adrian P Crawley, David Mikulis, and Karen D Davis. A cortical network sensitive to stimulus salience neutral behavioral context across multiple sensory modalities. Journal of neurophysiology. 87(1):615-635 620, 2002.
R. Becket Ebitz, Brianna Sleezer, Hank P. Jedema, Charles W. Bradberry, and Benjamin Y. Hay- den. Tonic exploration governs both flexibility and lapses. PLOS Computational Biology, 15(11); e1007475, November 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems, 2021.
Sebastian Flennerhag, Jane X Wang, Pablo Sprechmann, Francesco Visin, Alexandre Galashov, Steven Kapturowski, Diana L Borsa, Nicolas Heess, Andre Barreto, and Razvan Pascanu. Tem- poral difference uncertainties as a signal for exploration. arXiv preprint arXiv:2010.02255, 2020.
Samuel J. Gershman. Deconstructing the human algorithms for exploration. Cognition, 173:34-42, 2018. ISSN 0010-0277.
Samuel J. Gershman and Bastian Greshake Tzovaras. Dopaminergic genes are associated with both directed and random exploration. bioRxiv, 2018.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv: 16/ 1075077 2016.
Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020.
URL httpp://gii thw.///ggttut..ooo// com/ deepmind/ dm- haiku
Matteo Hessel, David Budden, Fabio Viola, Mihaela Rosca, Eren Sezener, and Tom Hennigan.
Optax: Composable gradient transformation and optimisation, JAX!, 2020. URL http:// g
11

---

Published as a conference paper at ICLR 2022
T. T. Hills, P. M. Todd, D. Lazer, A. D. Redish, and I. Couzin. Exploration versus exploitation in space, mind, and society. Trends Cogn Sci, 19(1):46-54 54,,,,544 Jan 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1997.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.0090 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. arXiv preprint arXiv:1605.09674, 2016.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv 2016.
preprint
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11(,), 2010.
Yuu Jinnai, Jee Won Park, David Abel, and George Konidaris. Discovering options for exploration by minimizing cover time. In International Conference on Machine Learning, pp. 3130-3139. PMLR, 2019a.
Yuu Jinnai, Jee Won Park, Marlos C Machado, and George Konidaris. Exploration in reinforcement learning with deep covering options. In International Conference on Learning Representations, 2019b.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent perience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2019.
J. M. Kembro, M. Lihoreau, J. Garriga, E. P. Raposo, and F. Bartumeus. Bumblebees learn foraging routes through exploitation-explrraaee cycles. R Soc Interface, 16(156):20191033, 07 2019.
Diederík Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412. 69900, 2014.
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A universal agent- centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume I, pp. 128-135. IEBE, 2005.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Cam Linke, Nadia M. Ady, Martha White, Thomas Degris, and Adam White. Adapting behaviour via intrinsic reward: survey and empirical study, 2019.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Daniel Mankowitz, Timothy Arthur Mann, and Shie Mannor. Adaptive skills adaptive partitions (ASAP). In Neural Information Processing Systems, 2016.
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Andrei A. Rusu. Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529533, 2015.
Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, and Michael Arbel. Deep reinforcement learn- ing with dynamic optimism. arXiv preprint arXiv:2/02.03765, 2021.
Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos. Count-based explo- ration with neural density models. CoRR, fts///73300310, 2017.
12

---

Published as a conference paper at ICLR 2022
Pierre- Yves Oudeyer and Frederie Kaplan. What is intrinsic motivation? typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Thomas G Power. Play and exploration in children and animals. Psychology Press, 1999.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay, In International Conference on Learning Representations, Puerto Rico, 2016.
Tom Schaul, Diana Borsa, David Ding, David Szepesvari, Georg Ostrovski, Will Dabney, and Simon Osindero. Adapting behaviour for learning progress, 2019.
Tom Schaul, Georg Ostrovski, lurii Kemaev, and Diana Borsa. Return- based scaling: Yet another normalisation trick for deep RL. arXiv preprint arXiv:2105.05347. 2021.
Jürgen Schmidhuber. Curious model-building control systems. In Proc. international joint confer- ence on neural networks, pp. 1458-1463, 1991.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions Autonomous Mental Development, 2(3):230-247, 2010.
Eric Schulz, Rahul Bhui, Bradley C. Love, Bastien Brier, Michael T. Todd, and Samuel Gershman. Structured, uncertainty-driven exploration in real-world consumer choice. Proceedings of the National Academy of Sciences, 116(28):3303--13900 June 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 1 112(1-2):181-
211, 1999.
Andrea L Thomaz and Cynthia Breazeal. Experiments in socially guided exploration: Lessons learned in building robots that learn with and without human teachers. Connection Science, 20 (2-3):91-110, 2008.
Sebastian B Thrun. Efficient exploration in reinforcement learning, 1992.
Michel Tokic Adaptive e-greedy exploration in reinforcement learning based on value differences.
In Annual Conference on Artificial Intelligence, pp. 203-210. Springer, 2010.
Gina G Turrigiano and Sacha B Nelson. Homeostatic plasticity in the developing nervous system. Nature reviews neuroscience, 5(2):97-107, 2004.
James A. Waltz, Robert C. Wilson, Matthew A. Albrecht, Michael J. Frank, and James M. Gold. Differential effects of psychotic illness on directed and random exploration. Computational Psy- chiatry, 480) August 2020.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In Maria Florina Balcan and Kilian Q Weinberger (eds.), Proceedings of The 33rd International Conference on Muchine Leurning, vol- ume 48 of Proceedings of Machine Learning Research, pp. 1995-2003 New York, New York, USA, 20-22 Jun 2016. PMLR.
Marco A Wiering and Hado Van Hasselt. Ensemble algorithms in reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 38(4)990-936, 2008
M. Wolfe, K. R. Cave, and S. L. Franzel. Guided search: an alternative to the feature integration model for visual search. Exp Psychol Hum Percept Perform, 15(3):419-43) 1989.
Daochen Zha, Wenye Ma. Lei Yuan, Xia Hu, and Ji Liu. Rank the episodes: A simple approach for exploration in Representations, 2021.
environments, International Conference on Learning
13

---

Published as a conference paper at ICLR 2022
A DETAILED EXPERIMENTAL SETUP
A.1 ATARI ENVIRONMENT
We use selection of games from the widely used Atari Learning Environment (ALE, Bellemare configured to not expose the 'life-loss" signal, and use the full action set (I8 discrete actions) for all games (not the per-game reduced effective action spaces). We also use the sticky-action randomisation as in (Machado et al. 2018). Episodes time-out after 108k frames (i.e. 30 minutes of real-time game play).
et al. (2013).
Differently from most past Atari RL agents following DQN (Mnih et al., 2015). our agent uses the raw 210 x 160 RGB frames input to its value function (one time, without frame stacking), though still applies a max-pool operation over the most recent 2 frames to mitigate flickering inherent to the Atari simulator. As in most past work, an action-repeat of 4 is applied, over which rewards are summed.
A.2 AGENT
The agent used in our Atari experiments distributed implementation of value- and replay-based RL algorithm derived from the Recurrent Replay Distributed DQN (R2D2) architecture (Kaptur- owski et al. 2019). This system comprises of a fleet of 120 CPU-based actors (combined with a single TPU for batch inference) concurrently generating experience and feeding it to a distributed experience replay buffer, and single TPU-based learner randomly sampling batches of experience sequences from replay and performing updates of the recurrent value function by gradient descent on a suitable RL loss.
The value function is represented by convolutional torso feeding into linear layer, followed by a recurrent LSTM (Hochreiter & Schmidhuber 1997) core, whose output is processed by a further linear layer before finally being output via Dueling value head (Wang et al.) 2016). The exact parameterisation follows the slightly modified R2D2 presented by Dabney et al. (202 and [Schaul al. (2021). Refer Table [ full list of hyper-parameters. It is trained via stochastic gra- dient descent a multi-step TD loss (more precisely, a 5-step Q-learning loss) with the use of periodically updated target network (Mnih et al.] (015)) for bootstrap target computation, using mini-batches of sampled replay sequences. Replay sampling is performed using prioritized expe- rience replay (Schaul al. 2016) with priorities computed from sequences' TD errors following the scheme introduced by Kapturowski et al.] (2019). As in R2D2, sequences of 80 observations are used for replay, with a prefix of 20 observations used for burn-in. In a slight deviation from the original, our agent uses fixed replay ratio of 1, i.e. the learner or actors get throttled dynamically the average number of times a sample gets replayed exceeds or falls below this value; this makes experiments more reproducible and stable.
Actors periodically pull the most recent network parameters from the learner to be used their ex- ploratory policy. In addition to feeding the replay buffer, all actors periodically report their reward, discount and return histories to the learner, which then calculates running estimates of reward, dis- count and return statistics perform return-based scaling (Schaul et al. 202I). If applicable, the episodic returns from the actors are also sent to the non-stationary bandit(s) that adapt the distribu- tion over exploration parameters (e.g., target ratios p or period lengths xxx). In return, the bandit(s)
provide samples from that distribution to each actor at the start of new episode, just like Schaul al. (019)).
Our agent is implemented with JAX (Bradbury et al. 2018), uses the Haiku (Hennigan et al. 2020), Optax (Budden et al., 2020b), Chex (Budden al. 2020a). and RLax (Hessel et 2022)) libraries for neural networks, optimisation, testing, and RL. losses, respectively, and Reverb (Cassirer et al.) 2020) for distributed experience replay.
A.3 TRAINING AND EVALUATION PROTOCOLS
All experiments for 200k learner updates. With a replay ratio of sequence length of 80 (adjacent sequences overlapping by 40 observations), a batch size 64, and an action-repeat this corresponds training budget of 200000 64 x 40 x x 4 ss 2B environment frames (which
14

---

Published as a conference paper at ICLR 2022
Neural Network
Convolutional torso channels
32, 64, 128, 128
Convolutional torso kernel sizes
7, 5, 5,3
Convolutional torso strides
4, 2, 2,
Pre-LSTM linear layer units
512
LSTM hidden units
512
Post-LSTM linear layer units
256
Dueling value head units
2 x 256 (separate linear layer for each of value and advantage)
Acting
Initial random No-Ops
None Yes (prob 0.25) 4
Sticky actions
Action repeats
Number of actors
120 400 environment steps
Actor parameter update interval
Replay
Replay sequence length
80 (+ prefix of 20 of burn-in)
Replay buffer size
10 observations (106 part-overlapping sequences)
Priority exponent
0.9
Importance sampling exponent
0.6
Fixed replay ratio
update per sample (on average)
Learning
Multi-step Q-learning
k 5
Off-policy corrections
None 0.997
Discount ?
Reward clipping
None
Return-based scaling
as in (Schaul et 2021
Mini-batch size
64
Optimizer settings
Adam Kingmaa Ba 2014).
learning rate 17 = 10
10
momentum 81 0.9, second moment 33 = 0.999
Gradient norm clipping
40
Target network update interval
400 updates
RND settings
Convolutional torso channels Convolutional torso kernel sizes
32, 64, 64
8, 4,3
Convolutional torso strides
4, 2, 1
MLP hidden units
128
Image downsampling stride
2 x 2
Table 1: Hyper-parameters and settings.
less than 10% of the original R2D2 budget). In wall-clock-time, one such experiment takes about 12 hours using 2 TPUs (one for the batch inference, the other for the learner) and 120 CPUs.
For evaluation, a separate actor (not feeding the replay buffer) is running alongside the agent using greedy policy (s = 0), and pulling the most recent parameters at the beginning of each episode. We follow standard evaluation methodology for Atari, reporting mean and median scores as introduced in (Mnih et al.] 2015) (i.e. the episode returns are normalised so that O corre- sponds the score of uniformly random policy while 1 corresponds human performance), as well as the mean 'human-cappaa score which caps the per-game performance at human level. Error bars or shaded curves correspond the minimum and maximum values across these seeds.
A.4 RANDOM NETWORK DISTILLATION
The agent setup for the X2 experiments differs in a few ways from the default described above. First, separate network is trained via Random Network Distillation (RND, (Burda al.) (018)). which consists of a simple convnet with an MLP (no recurrence); for detailed settings, see RND section in Table The RND prediction network is updated jointly with the Q-value network, the same
15

---

Published as a conference paper at ICLR 2022
data. The intrinsic reward derived from the RND loss is pursued at the same discount y 0.997 as the external reward in G. The Q-value network is augmented with a second head that predicts the Q-values for the intrinsic reward; this branches off after the Post-LLTMM linear layer' (with 256), and is the same type of dueling head, using the same scale normalisation method (Schaul et al.) 202T]. In addition, the 5-step Q-learning is adapted to use a simple off-policy correction, namely trace-cutting on non-greedy actions (akin to Watkins Q(A) with À = 1), separately for each learning head | The X' policy is the greedy policy according the Q-values of the second head. Note that because of these differences in set-up, and especially because the second head can function as an auxiliary learning target, it may be misleading to compare X and Xy results head-to-heatd we recommend looking at how things change within one of these settings (across variants of intra- episodic exploration or the baselines), rather than between them.
A.5 HOMEOSTASIS
The role of the homeostasis mechanism is transform sequence of scalar signals Iy R (for 1 S f T) into sequence of binary switching decisions y/ E {0, 1] so that the average number of switches approximates desired target rate p. that is + E. y/ p. and high values of It correspond to higher probability of = 1. Furthermore, the decision at any point can only be based on the past signals Ti:t. One way achieve this to exponentiate æ (to turn it into positive number xt) and then set an adaptive threshold to determine when to switch. Algorithm [ describes how this is done in pseudo-code. The implementation defines a time-scale of interest T == min(t, 100/ p). and uses it to track moving averages of three quantities, namely the mean and variance of as well as the mean of x+
Algorithm Homeostasis
Require: target rate p
1: initialize F + 0, x² +- x t
2: for E {1, (1,...,7) do
3: obtain next scalar signal return Zt.
4: set time-scale +- min(t, 100
P 5: update moving average E + (1 1)
6: update moving variance x² - f)x² (tz x)
7: standardise and exponentiate x + exp
カ--
8: update transformed moving average IF +
)
1
9: sample y/ Bernoulli (min (1, xt
z
10:
end for
In our informed trigger experiments we value promise as the particular choice of trigger signal Ie D promise (t k, t). As discussed in Section 3.1 when using bandit, its choices for target rates are p E (0.1, 0.001, 0.00013.
B OTHER VARIANTS
The results report in the main paper but subset of the possible variants that could be tried in this rather large design space. In fact, we have done initial investigations on a few of these, which report below.
B. ADDITIONAL EXPLORE MODES
Softer explore-exploit modes The all-or-nothing setting with greedy exploit mode and uni- form random explore mode clear and simple, but it is plausible that less extreme choices could work well too, such an s-greedy explore mode with 0.4 and an s-greedy exploit mode with
While this seemed like important aspect to it turned out to make very little difference in hindsight, see
Figure 15
16

---

Published as a conference paper at ICLR 2022
Tost
here
DORECMITS Ieverge
15000
2004
0.66
6.2
030
00.
130
135
325
330
6.5
669
6.56
808
625
100
120
12/00
ThE pACTINT
phzen's
30055
day gurber
20000
13:800
19200
3 Intrra intimed promceup
25 at intormat adienuphi
5500
750
10 011 informal (ancapn))
strerstrpp.
Leaneer CASS
1d
LeNTTER CWEE
Figure 8: Preliminary results comparing different informed triggers: value-discrepancy, action- mismatch, and variance-based, when using X, exploration mode.
E = 0.1. We denote this pairing as Xs. Preliminary results (see Figure 16) indicate that overall per- formance is mostly similar to Nt. possibly less affected by the choice of granularity and triggers.
Different discounts Another category of explore mode (X,) is to pursue external reward but at a different time-scale (e.g., a much shorter discount like y 0.97). This results in less of switch between explore and exploit modes, but rather in an alternation of long-term and short-term reward pursuits, producing a different kind of behavioural diversity. So far, we do not have conclusive results to report with this mode.
B.2 ADDITIONAL. INFORMED TRIGGERS
Action-mismatcc--brre triggers Another type of informed trigger to derive an uncertainty estimate from the discrepancies across ensemble. For example, an identical Q-learning update but are initialised differently. From that, we can measure multiple forms of discrepancy, nice and robust one is rank the actions according to each head and compute how large the overlap among the top-k actions is.
train two heads that
Variance-hased triggers Another type of informed trigger is measure the variance of the Q- values themselves, taken across such an ensemble (of two heads) and use that as an alternative uncertainty-based trigger.
Figure g shows preliminary results how performance compares across these two new informed triggers, in relation to the value-promise one from Section 2.2 Overall, the action-mismatch trigger seems have edge, at least in this setting, and plan to investigate this further in the future. From other probing experiments, it appears that for other explore modes, different trigger signals more suitable.
C ADDITIONAL RESULTS
This section includes additional results. Wherever the main figures included subset of games variants (Figures 5 we show full results here (Figures 11
respectively). and the aggre- are split into individual games Figure TO Also, some of the learning curves from Figures 4 [T] are shown in Figure 16 In addition. Figure 14] how the internal bandit probabilities evolve over time based starting mode for the experiments shown in Figure 6 Lastly, we provide some bonus illustrations for the intra-episodic design decisions in- cluded in the namings of our variants Figure with the hope of facilitating the interpretation and intuition for intra-episodic variants and their resulting behaviours.
12 13
gated performances of Figure are
17

---

Published as a conference paper at ICLR 2022
explore mode X
first mode explore
XI-intra (=, blind, n* x)
exploration exploitation periods last just
long because switching symmetric (=)
trigger exploration blindly according bandit (*) over number steps (n)
Figure 9: Extra example illustrating space design decisions for intra-episodic exploration.
frostbite
graviar
here
montezuma revenge
3000
1000
300
Loece
1200
2004
1000
2002
535 139 275 130
6.5
325
130
650 135 150
1080
Ep004t m/0m
splesdeeleeeel
XU-experiment-leeeee
XU level-0.00
XU-experiment-ler
2002
2000
XU intra/10.infrrmoddp""
LOWTEr OADE
LeaTeer CADE
Leanerr Cl2c
frostbite
gravitar
hero
montezuma revenge
200
750
1000
150
300
160
350
1300
75000
1820
1000
XI 0-episode-level-
1800
expertmenntievvi.ccc
15200
920
Xi-tta--eeei-Xii
280
3050
X0 D--ttraf10jnfommedd
X23
120 130 579 120
Figure 10: Extension of Figure showing performance results the mean episode return (with enror bars spanning the min and max performance across 3 seeds) for the 7 Atari games tried. We compare the four levels of exploration granularity described in section 2or Xy mode (top two rows) and XG mode (bottom two rows). Intra-episodic switching (red curve) is superior or compara- ble existing, monolithic well-established approaches. Note that here compare with the same intra-episodic switching mechanism (i.e., with XU- or XT- i in ra (1 0, i in formed d, p*, X) ), but there is at least one intra-episodic variant for each game which results in clear performance gains (just that is not the same intra-cpisodic variant across games or as the one shown here).
18

---

Published as a conference paper at ICLR 2022
hrattbite
grastirr
her
motoraume revenge
8
0.0
0.5
0.2
2
10-3
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.A
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
froutbete
grav Ear
here
montecuma tevangee
30000
6000
Hum
30000
2000
20000
4000
20000
1500
20000
2.000
10000
pacmman
phoemix
gurmer
12500
100000
You moddl inttoneettrt
/5,000
50000
25000
Val ronett al esturt
Xu-- mndil edped.jt
froatbite
3
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
me. pecmen
star: guiner
10000
6000
20000
1500
8000
4000
15000
Tpante
100000
5000
500
Alsphede caci
10000
7500
100000
Mettri intarmedd
3000
2900
100000
50000
ai (AQU))
Figure 11: Extension of figure all Atari games and intra-episodic switching variants tried. We show the characteristic space of exploration (summarized by rmedx and px on the X and Y axis, respectively) on rows 1, 5, and 6, and how different explore-exploit proportions translate to performance (error bars spanning the min and max performance across 3 seeds) on rows 3, 4, 7, and 8, for X' mode (top) and X2 mode (bottom). Note how different intra-episodic switching variants cover different parts of characteristic space and how the meta-controller adapts and changes the exploration statistics over time. This figure shows how fine-grained and varied intra-episodic switching can be, and how it translates to rich, diverse, and beneficial behaviours.
19

---

Published as conference paper at ICLR 2022
X
10 langui 0001
1000
G)
Xü-neratio
Xt irformed rate- 0,00
horfomoeeoee..o........ ratesi 001
grawitar
1000
Ausintrat11..iiidd. haro, target rata- rata-1000 1000
informed hero, target rate-O 0.01
infooommeddd... hero. target rata 20.001
hero, target rata-100 100
XU-intra(1D rt
Xü-tntrat10 ainnint
intratl informeed.p
XU intrat informes.p"
monteruum revenge. target rate 100 on monteoume_ revenge. target rate: 1000 on montezume, revenge, target ratem 0.001 on montenums, revenge, target ratee:O
Xu istnd on ms. pacman. rarget 1ote-100
XU Stind, 6) ms. patman, target 10te-1000
xu (arroio..rfommdppo pn os pacman, target rata-O 01
Xu on ms pacman. target rate-O
XU-intre(10 biindcr phownix, target ratem ion
XU-Intra110 D.lindd (i) on phoenis,, larget natee
XU Intretr Cetrei1..iffomrrr Dirfommdddd on phoenis, target ratemti.i
xu intred inffoommeed p puugs phoemix, target natern
iu birdd G) target rete-100
1nniiiar10..in..iii i) gurner. target
xú intrai1o intaiii...rfommett informed star punner. target rete--0
xú Cettraaiiiifoommst ph gunner target
Figure 12: Extension of Figure 5 to the 7 Atari games experimented with. First two columns: temporal structures for a blind, step-based trigger; the episodes we randomly selected correspond to 100 and 1000 fixed switching steps: the exploration period was fixed 10 steps. Last two columns: temporal structures obtained with an equivalent informed trigger and corresponding to target rates of 0.01 and 0.001, respectively.
20

---

Published as a conference paper at ICLR 2022
Ieral Ind 100 foothbe
Intarr Mind 0:53,6 foothiie
forthbe
X0-tttra and 120,6)
XDArtrule on guoellrr
Al aaind 22.50 on gracter
graettrr
XU-trzzle
on
Xikintrale sirdd (LG
Epiredereturr
Rinntummatr peanen
313-rEms) 3id 01 d mL pacmin
R0erbd-sMrdi335 mislouts
21o013
XW-rbrge bis 01 C m0a 9011
3ar gpmme
607
67
LOWnner 34
Figure 13: Extension of Figure 7 showing behavioural characteristics (exploration proportion px) between two forms of blind switching, stee-based (left) and probabilistic (center), with their corre- sponding performances (right).
21

---

Published as a conference paper at ICLR 2022
TC42200
geviter
peoras
plaee
3330
6:35
5.00
lever
Aetor
gravter
bern
rortecrs maveage
mT oscres
Precs
str gaweer
330
3200
022
Voslate
3330
0.00
groviter
ter
Portacurra reuenge
Pieess
iisz guiterr
Figure 14: Extension of Figure 6 showing the performance differences between two blind intra- episode experiments, starting either in explore (A', rows 2 and 4) or in exploit mode (G, rows and 3). We show the bandit arm probabilities for each of the step sizes nxx and how they change over the course of learning for Ny (top two rows) and for X' modes (bottom two rows). Findings: for symmetric blind triggers, starting with exploitation results in slower rates of switching (high n. = ng like red and green); in contrast, starting with exploration results in behaviours promoting higher switching rates (small nx = ng like blue and orange). Note that these preferences are not matching perfectly across all games, and thus results are domain-dependent.
XHiinra(=,.bind.p__5
frostbite
gravitr
hero
montezuma revenge
1150
10
1950
2-
120
3500
K25 137 075
180
130 13"
4:5
4.5
475
-20
13
130
130
127 125
130
ms. pac_pamman
phoenix
star gunner
122200
uspo
7000
off-pelicy without off-policy correction
18:00
CO5 137 020 180
150 1.5
6.5
120 +20 1X.
636 C75 220
100
Figure 15: Results of probe experiment around off-policy correction in X' mode. Red is Watkins Q(1) with À while blue uncorrected k-step Q-learning, in each case with returns of length at most & = 5. The performance is the same (or even slightly better) without off off-policy correction, showing that is critical in our current setting.
22

---

Published as a conference paper at ICLR 2022
Exploration mode ur frrmoom candom (XU)
Exploration mode based soft spsilons (85) Exploration mode bised on intirsic reward
trosttie
E:olode reTurT
Eskods
sodedoves
2.00
Wb nimi Misa
Ra stvile Min mose
Egi0200 14tur
1600
hdro
Tero
Ephoccte returm
8000
Epiodda ratum
mhN permmn mam
8006
8600
Eoleade returrs
400
proonix
E310000 000677
5000
Eptodde
gunner
gunner
gunner
Ep soda ctuu
Esp scoddo a002re
8000
1000
500
100
Figure 16: Comparing 3 different X' modes on the same 4 experimental settings and across 7 Atari games: uniform exploration (Ky, left), soft-epsilon-based exploration (K's, center), and intrinsic exploration (X,, right).
23


Reviews:
[{"review_id": "e3xcQZnyuyt", "reviewer": "Reviewer_ZCJR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is very strong, well-motivated, and empirically sound. Most of my concerns are with clarifying details of the experiments and improving the exposition. These are listed below:\n\n- paragraph 2 of the introduction: the typical answer to \"when to explore\" is when the agent is unfamiliar with the environment or its structure, i.e., early in its interactions with the environment. I would highlight why the heuristic of \"explore earlier\" should be challenged here instead.\n\n- paragraph 3 of the introduction: it's not clear what the connection to schizophrenia here is, other than that this was a study of explore vs. exploit; are these individuals somehow impaired in choosing between these options, etc.?\n\n- paragraph 1 of section 2 (methods): not sure that the example of riding a bicycle works - is the targeted acquisition of a new skill really exploration?\n\n- section 2.1, description of \"episode-level\": it would be more useful to define \"episode\" in the context studied here, rather than using the example of training games vs. tournament matches.\n\n- caption to Figure 1: the differences for D-G are unclear just by looking at this figure and the caption, other than that they are different intra-episode approaches; it would be better to clarify this\n\n- section 2.2, description of \"blind switching\": this refers to \"fractional episode length\", but if we don't get to choose the length of an episode, how do we implement fractional episode length ex-ante?\n\n- section 2.3, description of \"bandit adaptation\": the citations should be in-text citations\n\n- section 3: it would be useful to explain the observed differences between these domains; are there hypotheses for why certain domains are better suited for different switching mechanisms? The discussion in Section 3.3 starts to get at this, but does not specifically address why the differences may arise.\n\n- section 3.2: in appendix A.3, the compute budget is mentioned as being 2B frames; please clarify this difference\n\n- section A.1, regarding using the full action sets: does this mean that some games have meaningless actions, i.e., actions that cause no effect in the world? how does this affect learning and why was this choice made?\n\n- section A.1, regarding no life-loss signal: what exactly is an episode here if there is no life-loss signal? Is it just 108,000 frames (and why this number?) How does scoring work?\n\n- section A.1: why was the choice to use raw, unprocessed frames made? in particular, what is the effect of keeping color information for learning?\n\n- section A.2: is 1 TPU used (as mentioned here) or 2 TPUs (as mentioned in Section A.3)?\n\n- section A.2: citations should be in-text citations; \",\" should be \";\" after Schaul et al. (2021) in paragraph 2, line 4.\n\n- section A.3: \"while 2 TPUs\" should be \"using 2 TPUs\".\n\n- The captions to Figures 9 - 15, especially 9 and 10, should be expanded to clarify what the reader can learn from this figure (e.g., any hypotheses on what accounts for the differences between different environments)", "summary_of_the_paper": "This paper studies switching between exploit and explore modes in reinforcement learning. It discusses switching mechanisms based on time (\"blind switching\") and based on state (\"informed switching\"). Studying seven Atari games, an empirical analysis of different switching mechanisms is performed.", "summary_of_the_review": "The paper studies the relatively under-explored question of when agents should explore, introduces a novel exploration trigger called \"value promise discrepancy\", and performs a thorough empirical analysis in the domain of seven Atari games. Please see the main review for detailed comments and suggestions for improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W6v6g6zFQHi", "reviewer": "Reviewer_AkpW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of mode-switching is interesting. However, the presentation of the motivation as well as the results seem somewhat weak in my view. The main comparison baseline for the mode-switching architecture is the monolithic variant, where typically sparse rewards in hard exploration tasks are augmented with intrinsic reward signals. This means that the modes that they switch between in this paper are merged homogeneously in time. A valid problem that the authors point out for the monolithic case is that the scale of the intrinsic reward signal has to be tuned and may need to change in time. But there is no comparison to these methods and the superiority of their method to the monolithic variants are not highlighted well enough.\n\nThe authors try to circumvent the performance comparisons with other baselines by saying that they can obtain more diverse behaviours (in terms of exploration strategies?) and say that they don’t aim to show improved performance. However, this diversity argumentation is not strong enough in my view. First, it is not clear to me if the authors want to focus on the diversity of behaviour that is obtained within one variant of the mode-switching, e.g. informed switching, or whether they want to highlight the diversity across different granularities and switching mechanisms. If it is the latter, the diverse behaviour doesn’t necessarily translate into performance for the different games. And best performance for different games might be obtained with different strategies, but since this strategy is fixed prior to the experiment, it is not clear to me how this helps the diversity argument of the method itself. The authors point out that Montezuma’s Revenge and Phoenix have their best performance with different mode-switching behaviours, but these results have a really high variance. Perhaps, 3 seeds are just not enough and more seeds are needed to draw reliable conclusions.", "summary_of_the_paper": "This paper proposes a mode-switching strategy for the exploration/exploitation dilemma instead of monolithic behaviour policies in order to obtain more diverse behaviour. Different granularities for the timing of the switches as well as different switching mechanisms are investigated (blind vs. informed switching). The focus for exploration is not on how, but when. For exploration, they both use Random Network Distillation (RND) as well as a uniform policy. Their experiments are conducted on the Atari Learning Environment (ALE), where they provide performance and diversity results.", "summary_of_the_review": "I believe that the idea presented in this paper is interesting but the results are lacking. The related work section briefly covers some similar methods, and I think comparisons are still needed with these other methods. E.g. GoExplore also focuses on the *when* question of exploration and should be included as a baseline as well as works with monolithic behaviour policies where the mode-switching is replaced by a weighting problem of external and intrinsic rewards of a single behaviour policy.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Qcv_GiwGPhr", "reviewer": "Reviewer_TbGe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper conducts a series of experiments aimed at answering the question of when we should switch between exploitation and exploration during RL learning.\nAs positive points we can cite the wide related bibliography, from analogies with the animal system (human included) of exploring to other techniques such as the use of options. Another point to emphasize is the clear, careful and pleasurable writing that the authors developed in the paper. The authors managed to transmit a moment of reflection and expansion of thoughts about the possible behaviors shown by agents who learn with RL. Moments of reflection in an area that transforms at a very high speed are always welcome.\n\nPerhaps the negative point is the limitation of the contribution -- since it is still a study that must be deepened in order to provide effective and efficient guidelines for RL system developers working in real applications.\n\nHowever, it is an in-depth study, with interesting results. The article is worth being divulged to remind everyone working in the RL area that there is still a lot to study, investigate and evaluate in order to have robust, efficient and effective systems.", "summary_of_the_paper": "This paper investigates when to switch between exploitation and exploration and how long to stay in each exploration mode during RL learning. It proposes new ways to explore the subject, especially with intra-episodic exploration variants. It presents a large body of study results (10 pages of appendices!!), and concludes with very thought-provoking suggestions and discussions.", "summary_of_the_review": "The paper brings an in-depth study, with interesting results and very well written. It is worth being divulged to remind everyone who works in the RL area that there is still a lot to study, investigate and evaluate in order to have robust, efficient and effective systems.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fjc2fBjmhwZ", "reviewer": "Reviewer_3t1W", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- this paper investigates a novel area, which seems very important at a high level.\n- it also proposes novel methods to start to address this area.\n\nWeaknesses:\n- the paper does not feel very focused. There are lots of different ideas and methods presented, but the takeaways are unclear. \n- the figures are a bit hard to parse \n\nFirst, I applaud the authors for tacking a new and unexplored area. However, the takeaways and next research steps are unclear. At least on Atari, the benefits of intra-episodic exploration seem limited. \n\nSome suggestions for improving the paper:\n- It would be helpful to have the different algorithm variants in Section 2 spelled out, especially regarding the different switching mechanisms. The current descriptions are quite high-level and it would be helpful to make them concrete.\n- The tuple notation (shown in Figure 2) is a bit hard to parse. It would be helpful to show several examples, so that the differences in the different elements of the list can be seen more clearly.\n- I think the paper would be a lot stronger if there were some tasks which more convincingly demonstrated the benefits of intra-episode exploration. This could be a new task which the authors design themselves. I think including the Atari experiments is useful in that it shows their methods can improve performance on a standard benchmark, however, the monolithic exploration methods already work quite well on these tasks, and it is not clear if there is that much more improvement to be had by exploring at a finer granularity. I do agree with the authors that in the “big picture”, monolithic exploration is likely suboptimal and more informed exploration will be necessary. I found their motivating example of learning to ride a bike while maintaining necessary daily activities to be very helpful. Can you design some task which distills this task in a simpler format? For example, some setup where the agent must regularly find food from a predictable source (exploiting) but also must explore when it can between finding food. Introducing new tasks which measure the ability to optimally switch between explore and exploit modes would also be useful to the community in building on this work. \n\n", "summary_of_the_paper": "This paper proposes to study exploration at different levels of granularity. Current methods either explore at the level of individual steps (e.g., \\epsilon-greedy), or at the level of experiments (e.g. first a reward-free exploration phase, followed by a task-dependent learning phase using the gathered data). This paper proposes to study exploration at the intra-episodic level, i.e. where the agent switches between exploration and exploitation within the same episode. \n\nThey discuss various design choices to perform exploration at this level, for example switching after a certain number of steps or with a certain probability, or switching based on the discrepancy between the predicted value and actual experienced value. \n\nThe experimental results show that including intra-episodic exploration gives a modest benefit over other exploration schemes when using an R2D2 base agent. Other insights are also included, which show that the proportion of exploration does change throughout the learning process, indicating that different degrees of exploration are useful at different stages. They also show that the informed switching component learns switching behaviors which are non-uniform throughout the episodes.", "summary_of_the_review": "I'm on the fence about accepting this paper. On one hand I think it is good that the authors are exploring a new and important area and the ideas are interesting. On the other hand, this work still feels preliminary and the benefits of intra-episode exploration are not yet convincingly demonstrated.  I am not strongly opposed to accepting this paper since it could at least be a starting point for research in this area. However I think that if the authors could introduce new tasks where intra-episodic exploration convincingly helps, then I think this would be a very strong submission to a later conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Exploration can happen at various levels of granularity and at different times during an episode,  and this work performs a study of the problem of exploration (when to explore/when to switch between exploring and exploitation, at what time-scale to do so, and what signals would be good triggers to switch). The study is performed on atari games.

Strenghts:
------------
The study is well motivated and the manuscript is overall well written
Studies a new problem area, and proposes an initial novel method for this problem
extensive study on atari problems

Weaknesses
--------------
some clarity issues as pointed out by the reviewers
no illustrative task is given to give a more intuitive exposition of the "when to explore" problem
comparison to some extra baselines like GoExplore would have been insightful

Rebuttal:
----------
Most clarity issues have been addressed satisfactorily. It has been explained why some requests for extra baselines would be challenging/or not relevant enough. While the authors agree that GoExplore would be an interesting baseline, they seem to have not added it. An illustrative task was not provided.

Summary:
------------
All reviewers agree that this manuscript opens up and tackles a novel direction in exploration, and provides an extensive empirical study on atari games (a standard benchmark for such problem settings). While I agree with the reviewers that point out that this paper could have been made stronger by adding an illustrative task and additional baselines like GoExplore, there is a general consensus that the provided empirical study on this novel problem setting is a good contribution in itself. Because of this I recommend accept.
Decision: Accept (Spotlight)