When should agents explore?
Authors: Miruna Pislar
David Szepesvari
Georg Ostrovski
Diana L Borsa
Tom Schaul
Venue: NeurIPS 2021 Submitted (2021)
Keywords: exploration
mode-switching
reinforcement learning
Atari

Abstract: Exploration remains a central challenge for reinforcement learning (RL). Virtually all existing methods share the feature of a *monolithic* behaviour policy that changes only gradually (at best). In contrast, the exploratory behaviours of animals and humans exhibit a rich diversity, namely including forms of *switching* between modes. This paper presents an initial study of mode-switching, non-monolithic exploration for RL. We investigate different modes to switch between, at what timescales it makes sense to switch, and what signals make for good switching triggers. We also propose practical algorithmic components that make the switching mechanism adaptive and robust, which enables flexibility without an accompanying hyper-parameter-tuning burden. Finally, we report a promising initial study on Atari, using two-mode exploration and switching at sub-episodic time-scales. 

When should agents explore?
Anonymous Author(s)
Affiliation Address email
Abstract
Exploration remains central challenge for reinforcement learning (RL). Virtually all existing methods share the feature of a monolithic behaviour policy that changes only gradually (at best). In contrast, the exploratory behaviours of animals and hu- mans exhibit a rich diversity, namely including forms of switching between modes. This paper presents an initial study of mode-switching. non-monolithic exploration for RL. We investigate different modes to switch between, at what timescales it makes sense switch, and what signals make for good switching triggers. We also propose practical algorithmic components that make the switching mechanism adaptive and robust, which enables flexibility without an accompanying hyper- parameter-tumm tuning burden. Finally, we report promising and detailed analysis on Atari, using two-mode exploration and switching at sub-episodic time-scales.
2
11
Introduction
12
The trade-off between exploration and exploitation is described the crux of learning and behaviour across many domains, not just reinforcement learning [Sutton and Barto, 2018]. but also in decision making [Cohen et al., 2007], evolutionary biology [Cremer et al., 2019]. ecology [Kembro al., 2019], neuroscience (e.g., focused versus diffuse search in visual attention [Wolfe et al., 1989], dopamine regulations [Chakroun et al., 2020J), cognitive sciences [Hills al., 2015], well as psychology and psychiatry [Addicott et al., 2017]. In nutshell, exploration is about the balance between taking the familiar choice that is known to be rewarding and learning about unfamiliar options of uncertain reward, but which could ultimately be more valuable than the familiar options.
14
16
17
18
19
20
Ample literature has studied the question of how much to explore, that is how to set the overall trade-off (and how to adjust over the course learning) [Jaksch et al., 2010, Cappé et al., 2013, Lattimore and Szepesvári, 2020, Thrun, 1992], and the question of how explore, namely how to choose exploratory actions (e.g., randomly, optimistically, intrinsically motivated, or otherwise) [Schmidhuber, 1991, Oudeyer and Kaplan, 2009, Linke et al., 2019]. In contrast, the question of when 26 to explore has been studied very little, possibly because it does not arise in bandit problems, where a
22
24
27 lot of exploration methods are rooted. The 'when' question and its multiple facets are the subjects of this paper. We believe that addressing it could lead to more intentional forms of exploration.
28
Consider agent that has access to two modes of behaviour, 'explore' mode and an 'exploit' 30 mode (c.g., a random policy and greedy policy, as in e-greedy). Even when assuming that the overall proportion of exploratory steps is fixed, the agent still has multiple degrees of freedom: it can explore more at the beginning of training and less later phases; it may take single exploratory steps execute prolonged periods of exploration; it may prefer exploratory steps early or late within an episode; and it could trigger the onset (or end) of an exploratory period based on various criteria.
32
34
35 Animals and humans exhibit non-trivial behaviour in all of these dimensions, presumably encoding
36 useful inductive biases that way [Power, 1999]. Humans make use of multiple effective strategies,
37
such as selectively exploring options with high uncertainty (a form of directed, information-seeking exploration), and increasing the randomness of their choices when they are more uncertain [Gershman,
38
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

While the choice of behaviour in pure exploit mode is straightforward, namely the greedy pursuit of external reward (or best guess thereof), denoted by G. there are numerous viable choices for behaviour in pure explore mode (denoted by X). In this paper we consider two standard ones: 39 2018, Gershman and Tzovaras, 2018, Ebitz et al., 2019]. Monkeys use directed exploration to manage
40 explore-exploit trade-offs, and these signals are coded in motivational brain regions [Costa al..
41 2019]. Patients with schizophrenia register changes in directed exploration and experience low-grade
42 inflammation when shifting from exploitation to random exploration [Waltz et al., 2020, Cathomas
43 et al., 2021]. This diversity is what motivates us to study which of these can benefit RL. agents in turn,
44 by expanding the class of exploratory behaviours beyond the commonly used monolithic ones (where
modes are merged homogeneously in time).
2 Methods
47
The objective of an RL agent is to learn policy that maximises external reward. At the high level, it achieves this by interleaving two processes: generating new experience by interacting with the environment using a behaviour policy (exploration) and updating its policy using this experience (learning). As RL is applied to increasingly ambitious tasks, the challenge for exploration becomes 51 to keep producing diverse experience, because if something has not been encountered, it cannot be
48
49
50
52 learned. Our central argument therefore simple: monolithic, home-hooggeneuu behaviour policy
53 is strictly less diverse than a heterogeneous mode-switching one, and the former may hamstring
54 the agent's performance. As an illustrative example, consider a human learning how to ride a bike
55 (explore), while maintaining their usual happiness through food, sleep, work (exploit): there is a stark
56 contrast between a monolithic, -omooeenggs behaviour that interleaves twist of the handlebar or turn of pedal once every few minutes and the mode-switching behaviour that dedicates prolonged periods of time exclusively to acquiring the new skill of cycling.
58
59
2.1
Exploration modes
60
61
62
63
Xr, the naive uniform random policy, and X,, an intrinsically motivated behaviour that exclusively pursues a novelty measure based on random network distillation (RND, [Burda et al., 2018)). See Section& and Appendix B for additional possibilities of X. In this paper we choose fixed behaviours for these modes, and focus solely the question of when to switch between them. In our setting, overall proportion of exploratory steps (the how much), denoted by px, is not directly controlled but derives from the when.
64
65
67
2.2 Granularity
70
An exploration period is an uninterrupted sequence of steps explore mode. We consider four choices of temporal granularity for exploratory periods, also illustrated on Figure 1:
71
72
Step-level exploration is the most common scenario, where the decision to explore is taken indepen-
73
dently at each step, affecting one action.' The canonical example is E-greedy (Fig.
Experiment-level exploration is the other extreme, where all behaviour during training is produced in explore mode, and learning is off-policy (the greedy policy is only used for evaluation). This scenario is also very common, with most forms of intrinsic motivation falling into this category, namely pursuing reward with an intrinsic bonus throughout training I:ig.
75
76
77
78
Episode-level exploration is the case where the mode is fixed for an entire episode at a time (e.g.. training games versus toumnamet matches in sport), see 1ig. This has been investigated for simple cases, where the policy's level of stochasticity sampled at the beginning of each episode [Horgan et al., 2018, Kapturowski et al., 2019, Zha et al., 2021].
79
80
81
82
Intra-episodic exploration is what falls in- be-betwenn step- and episode-level exploration, where exploration periods last for multiple steps, but less than full episode. This is the least commonly studied scenario, and will form the bulk of our investigations 1Fig.1:D,E.F.C.
83
84
'The length of exploratory period tends to be short, but it can be greater than 1, multiple consecutive step-wise decisions to explore can create longer periods.
'Note that it is also possible to interpret e-greedy as experiment-lewe exploration, where the X' policy is fixed to noisy version of G.
2

---

episode boundaries
explore mode X'
exploit mode g
Intra eptsodic
exploration
A
B
G
C
PX
F DE
alep
medx
1 episoda
exparmmnt
full
G
Figure 1: Illustration of different types of temporal structure for two-mode exploration. Left: Each line A-G depicts an excerpt of an experiment (black lines show episode boundaries, experiment continues the right), with colour denoting the active mode (blue is exploit, magenta is explore). A is of experiment-level granularity, B episode-level, C step-level. and D-G are of intra-episodic exploration granularity. Right: The same examples, mapped onto a characteristic plot of summary statistics: overall exploratory proportion Px versus typical length of an exploratory period medx. The yellow-shadee area highlights the intra-episodic part of space studied this paper (some points are not realisable, e.g., when px R 1 then medx must be large). C, D, E, F share the same = 0.2, while interleaving exploration modes in different ways. D and E share the same medx value, and differ only whether exploration periods are spread out, or happen toward the end of episode.
85 We denote the length of an exploratory period by n.x (and similarly ng for exploit mode). To characterise granularity, our summary statistic of choice is medx median(nx)) Note that there are two possible units for these statistics: the raw steps or the proportion of the episode length L. The latter has different (relative) semantics, but may more appropriate when episode lengths vary widely across training. We denote it as rmed == mnedian(nxx/la
86
87
88
89
2.3
Switching for intra-episodic exploration
91
Granularity is but the coarsest facet of the 'when' question, but more precise intra-episode timings (when start and when to stop an exploratory period) are important aspects too.
93
Blind switching The simplest type of switching mechanism does not take state or time into account (thus we call it blind), and only concerned with producing switches at some desired time resolution. It can be implemented deterministically through a counter (e.g.. enter explore mode after 100 exploit mode steps), or probabilistically (e.g., each step, enter explore mode with probability 0.01). Its expected duration can be parameterised in terms of raw steps, in terms of fractional episode length. The opposite of blind switching is informed switching, as discussed Section 2.4.
94
95
96
96
39
Asymmetry In general, the mechanism for entering the explore mode can differ from the one for exiting it (to enter the exploit mode), and this is crucial to obtain flexible overall amounts of exploration switching were symmetric, the proportion would be px 0.5.
100
Starting mode When periods last for significant fraction of episode length, it also matters how the sequence initialised, i.e., whether an episode starts in explore or in exploit mode, or more generally, whether the agent explores more early in episode more later It conceivable that the best choice among these is domain dependent (see Figure 6): in most scenarios, the states at the beginning of episode have been visited many times, thus starting with exploit mode can be beneficial; in other domains however, early actions may disproportionately determine the available future paths (e.g., build orders in StarCraft [Churchill and Buro, 2011)).
103
105
107
108
109
2.4
Informed switching with triggers
Going beyond blind switching opens up another rich set of design choices. We decompose the mechanism into two parts. First, sealar trigger signal is produced by the agent at each step, based on its current information drawing inspiration from human behaviour, the triggering signal is intended
I11

---

113 to be proxy for uncertainty [Schulz et al.. 2019]. Second, a binary switching decision is taken based
114 on the trigger signal, for example by comparing threshold. Again, the type of trigger and its configuration will in general not be symmetric between entering and exiting exploratory period.
115
116
Value promise trigger To keep this paper focused, we will look at one such trigger, dubbed 'value promise discrepancy' (see Appendix B for additional competitive variants). This is an online proxy of how much of the reward that the agent's past value estimate promised (k steps ago) have actually come about. The intuition is that in uncertain parts of state space, this discrepancy will generally be larger than when everything goes as expected. Formally,
117
118
120
k-1
D promisse (t - t), =- V(S+-L) Ee'Reii-
1=0
121
where (s)) is the agent's value estimate at state 8, R is the reward, and y is discount factor.
122
Homeostasis In practice, the scales of trigger signals may vary substantially across domains, and across training time, for example, the magnitude of D Dromise will depend on reward scales and density, and can decrease over time as accuracy improves (the signals could also be noisy). This
125
124
125
means that naively setting a threshold paprr-paramtter is impractical. For a simple remedy, we have taken inspiration from neuroscience [Turrigiaaoo and Nelson, 2004] to add homeostasis the binary switching mechanism, which tracks recent values of the signal and adapts the threshold for switching so that a specific average targel rate is obtained. This functions as an adaptive threshold, making tuning straightforward because the target rate of switching can be configured independently of the scales of the trigger signal. See Appendix A for the details of the implementation.
126
128
129
130
131
2.5
Adaptation instead of tuning
132
Our approach introduces additional flexibility to the exploration process, even when holding the specifics of the learning algorithm and the exploration mode fixed. The two main added dimensions are when (or how often) to enter explore mode, and when (or how quickly) to exit it. To avoid this becoming a hyper-parameter tunigg burden, we propose to follow [Schaul et al., 2019] and [Badia et al., 2020a], and delegate the adaptation of these settings to meta-controller (implemented as non-stationary multi-armed bandit that maximises episodic return). As an added benefit, the "when' of exploration now become adaptive to both the task, and the stage of learning.
133
134
135
136
137
3
Results
139
140
The design space we propose contains a number of atypical ideas for how to structure exploration. For this reason, opted to keep the rest of our experimental setup very conventional, and include multiple comparable baselines, ablations and variations.
142
Setup: R2D2 on Atari We conduct our investigations on subset of games of the Atari Learning Environment [Bellemare et al., 2013], a common benchmark for the study of exploration. All experiments are conducted across 7 games (FROSTBITE, GRAVITAR, H.E.R. O., MONTEZUMA' REVENGE, Ms. PAC-MAN, PHOENIX, IX, STAR GUNNER), the first of which are classified as hard exploration games [Bellemare et al., 2016), using 3 seeds per game. For our agent, use the R2D2 architecture (Kapturowski et al., 2019], which is a modern, distributed version of DQN [Mnih et al., approximate its Q-value function. This is a common basis used in exploration studies, e.g.. [Dabney et al., 2020, Badia et al., 2020b,a]. The only major modification conventional R2D2 is its exploration mechanism, where instead we implement all the variants of mode-switching introduced in Section 2. Separately from the experience collected for learning, we run an evaluator process that assesses the performance of the current greedy policy. This is what we report in all our performance curves (see Appendix A for more details).
144
145
146
147
148
149
2015] that employs a recurrent network
150
151
152
153
154
155
There are few simple baselines worth comparing to, namely the pure explore mode (px 1, I:A) and the pure exploit mode (px 0). as well as the step-wise interleaved s-greedy execution 1Fi).:CC), where px = 0.01 = & (without additional episodic or intra-episodic structure). Given its wide adoption in well-tuned prior work, we expect the latter to perform well overall.
Baselines
157

---

explore mode Xy
first mode greedy
X0-intra (100, bl id, G)
exploration periods end after 100 steps
trigger exploration blindly according to bandit (*) over probabil (p)
Figure 2: Illustrating the space design decisions for intra-episodic exploration.
The fourth baseline picks mode for entire episode at time (Fig. with the probability of picking X' being adapted by a bandit meta-controller. We denote these as experiment-leves experiment-
180
161
step-level-O- 01 and spisode-level-- respectively. For each of these, we have a version with uniform (Ryy) and intrinsic (A2) explore mode.
182
163
3.1
Variants of intra-episodic exploration
184
As discussed in Section 2, there are multiple dimensions along which two-mode intra-episodic ex exploration can vary. The concrete ones for our experiments are:
185
166
Explore mode: uniform random Ny, or RND intrinsic reward Xy (denoted XU and XI).
167
Explore duration (nx)): this can be a fixed number of steps (1, 10, 100), or one of these is adaptively picked by bandit (denoted by *), the switching is symmetric between entering end exiting explore mode (denoted by =).
169
170
Trigger type: either blind or informed (based on value promise, see Section 2.4).
171
Exploit duration (ng): for blind triggers, the exploit duration can be parameterised by fixed number of steps (10, 100, 1000, 10000), indirectly defined by probability of terminating (0.1, 0.01,, 0.001, 0.0001), or adaptively picked by bandit over these choices (denoted by n* p*, respectively). For informed triggers, the exploit duration is indirectly parameterised by a target rate in (0.1, 0.01, 0.001, 0.0001). or a bandit over them (p*), which in turn transformed into adaptive switching threshold by homeostasis (Section 2.4).
173
175
Starting mode: 9 greedy (default) or X' explore (denoted by G or X).
178
We can concisely refer to a particular instance by a tuple that lists these choices. For example, KU-intraf (100, informed, p x)) denotes uniform random exploration Xy, with fixed 100-step explore periods, triggered by the value-promise signal at a bandit-determined rate, and starting in explore mode. See Figure 2 for an illustration.
179
180
181
182
3.2
Performance results
183
We start by reporting overall performance results, to reassure the reader that our method is viable (and convince them keep reading the more detailed and qualitative results in the following sections). Figure 3 shows performance across 7 Atari games according to two numan-normalised aggregation metrics (mean and median), comparing one form of intra-episodic exploration to all the baselines, separately for each explore mode (Xu and X2). The headline result is that intra-episodic exploration improves over both step-level and episode-level baselines (as well as the pure experiment- level modes that we would not expect to be very competitive). The full learning curves per game are found in the appendix, and show scores on hard exploration games like MONTEZUMA'S 'S REVENGE or PHOENIX that are also competitive in absolute terms (at our compute budget of 1B frames).
184
185
186
187
188
190
192
Note that there is a subtle difference the learning setups between Xy and X2, as the latter requires training a separate head to estimate intrinsic reward values. This is present even in pure exploit mode, where it acts as an auxiliary task only [Jaderberg al., 2016), hence the differences in pure greedy curves in Figure 3. For details, see Appendix A.
193
194
195
196
3.3
Diversity results
197
In study like ours, the emphasis is not on measuring raw performance, but rather on characterising the diversity of behaviours arising from the spectrum of proposed variants. A starting point is to
5

---

Meen normalled score
Median human-sormelized score
Mean numan-normalizee score
Median humat-normalizee score
2.02
XU experinent lovel
experiment lovelc
175
175
stay
1.5
1:32
introtte informoo, p130
1.25
2.25
0.75
0.75
0157
0.50
0.23
0:23
0.5 0.0
0.00
3.5
0.5 0.2 0.6 0.5
0.02
0.2
0.0
Figure 3: Human-normalized performance results aggregated over Atari games and 3 seeds, comparing the four levels of exploration granularity. Left two: uniform explore mode Ny. Right two: RND intrinsic reward explore mode Xy. In each case, the baselines are pure modes X' and G. step-level switching with e-greedy, and episodic switching (with bandit-adapted proportion). In each setting, intra-episodic exploration is on par or better than the baselines.
tostbrre
30"1
0.0
0.3
0.5
0.6
3.8
3.0
0.0
0.2
0.4
0.6
0.0
3.5
0.0
0.3
0.1
0.6
0.8
3.0
0.0
0.
0.1
0.5
0.5
19are
.00.0
8006
20000
grovitrr
mortcaurra rorenge
20"11
mesil
modd
imodi
cancentaaa reverge
18:000
6000
b00099
20 sode devers
200
30 Sotattat anaid))
3000
UCC300
Figure 4: Rows 1 and 3: Summary characteristics Px and rmedx of induced exploration behaviour, for different variants of intra-episodic exploration (and an episodic baseline for comparison), on subset of Atari games. Bandit adaptation can change these statistics over time, hence square and cross markers show averages over first and last 10% training, respectively. Rows and Corresponding final scores (averaged over final 10% of training). Error bars show the span between min and max performance across seeds. Note how different variants cover different parts of characteristic space, and how the bandit adaptation shifts the statistics into different directions for different games. See main text for further discussion of these results and Appendix C for other games and variants.
199
return Figure (right), and assess how much of the previously untouched space is now filled by intra-episodic variants, and how the 'when' characteristics translate into performance. Figure 4 answers these questions, and raises some new ones. First off, the raw amount of exploration px is not a sufficient predictor of performance, implying that the temporal structure matters. It also shows substantial bandit adaptation work: compare the exploration statistics at the start (squares) and end-point of training (crosses). and how these trajectories differ per game; a common pattern is
200
201
202
203
204
205
that reducing px far below 0.5 is needed for high performance. Interestingly, these adaptations are similar between X and Xr, despite very different explore modes (and differing performance results). would expect prolonged intrinsic exploration periods to more useful than prolonged random ones, and indeed, comparing the high-rmed, variant (purple) across Xo and X1, it appears more
207

---

XU nbrat10.olind.e
XU introlze, inw.mmooooo.....m.m (9.8) evol-frestbine
level trastbite
37500
15200
2500
2000
200
Intrato,, b.
X0 1n intomee pe.Gi
200
403
003
303
3000
1300
3400
250
500
750
1030
1250
1500
00
0.5
10
15
tphorde Stggo
tpisode 3tops
Leamer stops
XU-intraf1e, blinddp
informed
lnvelmgraviar
level- level-grvv cell-graaaaa tar
Unewier
5010
Esteode retur
4320
0010
intintrat biidd prd.....co
Xilintral informed pt.
Pycncate
Fpoezsle
105
Figure 5: Left and center:
Illustraioon of detailed temporal structure within individual episodes, on FROSTBITE (top) and GRAVITAR (bottom), contrasting two trigger mechanisms. Each subplot shows 15 randomly selected episodes (one per row) that share the same overall exploration amount Px = 0.1. Each vertical bar (magenta) represents an exploration period of fixed length π1,r 10; each blue chunk represents an exploitation period. Left: blind, step-based trigger leads to equally spaced exploration periods. Center: a trigger signal informed by value promise leads to very different within-episode patterns, with some parts being densely explored, and others remaining in exploit mode for very long. Right: the corresponding learning curves show clear performance benefit for the informed trigger variant (orange) in this particular setting. Appendix has similar plots for many more variants and games.
frostbite
trostbite
star guehe
gravitar
star ggungerrrrrrrr
eevitar
XII-intrapp hwn.iiiaappppiippp.... blind,pt.cc [..
X-intral blind,
hu.intralp.bilnd.p...
nt-intral plidd,
phoen
ero
phoon
ero
pacmenn mcatezuma, revenge
paw.aa.
mouteruma revenge
Figure 6: Starting mode effect. Final mean episode return for two blind intra-episode experiments that differ only in start mode, greedy (blue) or explore (orange). Scores are normalised so that 1 is the maximum result across the two start modes. Either choice can reliably boost or harm performance, depending on the game. Left: uniform explore mode Xy. Right: intrinsic reward explore mode Xi.
209
beneficial for the latter. Zooming in on specific games, a few results stand out: in Ny mode, the only variant that escapes the inherent local optimum of PHOENIX is the blind, doubly adaptive one (purple), with the bandits radically shifting the exploration statistics over the course training. In contrast, the best results on MONTEZUMA'S REVENGE are produced by the symmetric trigger variant (blue), which is forced to retain high Px- Finally, FROSTBITE is the one game where an informed
211
213
trigger (red) clearly outperforms its blind equivalent (purple).
215 These insights are still limited to summary statistics, so Figure looks in more depth the detailed temporal structure within episodes (as in Figure 1, left). Here the main comparison is between blind
7

---

XU intrate 100 G) on frestbite
XIU-intra| wwnd.dddddd. O (1, oe frostb
Rostile
8000
8000
0.00
1000
Sphuider relurr
6000
8000
2000
2000
3000
XU-intrat- 2ww.d........... Gl prob basod
XU-ntrrat Sotttrat--ttiddddd. 100, stap based
0.6
0.0
0.8
033
000
1:0 12t Lear steps
330
1.1
Figure 7: Left and center: Contrasting the behavioural characteristics between two forms of blind switching, step-based (left) and probabilistic (center), the example of FROSTBITE. Each point is an actor episode, with colour indicating time in training (blue for early, red for late). Note the higher diversity of p.x' when switching probabilistically. Right: Corresponding performance curves indicate that the probabilistic switching (red) has a performance benefit, possibly because it creates the opportunity for 'lucky' episodes with much less randomness in a game where random actions can easily kill the agent. For more games, please see the Appendix C.
217
and informed triggers, illustrating that the characteristics of the fine-grained within-episode structure differ massively, despite attaining the same high-level statisties px and medx. We can see quite a lot of variation in the trigger structure - the moments we enter exploration are not evenly spaced anymore. As bonus, the less rigid structure of the informed trigger (and possibly the more carefully chosen switch points) end up producing better performance too.
219
220
221
complementary dimension, differentiating the effects of starting in explore or exploit mode. In brief, each of these can be consistently beneficial in some games, and consistently harmful in others. Another observation here is the dynamics of the bandit adaptation: when starting in exploit mode, it exhibits a preference for long initial exploit periods in many games (up to 10000 steps), but that effect vanishes when starting in explore mode (see also Appendix C). More subtle effects arise from the choice of parameterisation of switching rates. Figure 7 shows a stark qualitative difference on how probabilistic switching differs from step-count based switching, with the former spanning a much wider diversity of outcomes, which improves performance.
Figure 6 sheds light
223
225
226
227
228
229
3.4 Take-aways
231
Summarising the empirical results in this section, two messages stand out. First, there seems to be a sweet spot in terms of temporal granularity, and intra-episodic exploration is the right step towards finding it. Second, the vastly increased design space of our proposed family of methods gives rise to large diversity of behavioural characteristics; and this diversity is not superficial, it also translates to meaningful performance differences, with different effects in different games, which cannot be reduced to simplistic metrics, such as px. In addition, we provide some sensible rules-of-thumb for practitioners willing to join us on the journey of intra-episodic exploration. In general, it is useful to
232
233
235
236
237
let bandit figure out the precise settings, but it is worth curating its choices to most handful. 238
239
Jointly using two bandits across factored dimensions is very adaptive, but can sometimes be harmful when they decrease the signal-to-noise ratio in each other's 's learning signal. Finally, the choice of the uncertainty- based trigger should be informed by the switching modes (see Appendix B for details).
240
242 4 Discussion
Time-based exploration control The emphasis of our paper has been on the potential benefits of heterogeneous temporal structure in mode-switching exploration. But there is another, more mundane potential advantage over monolithic approaches: it may be easier and more natural to tune 244
246
related to an explicit exploration budget (c.g., via px) than to tune an intrinsic reward coefficient, especially if extrinsic reward scales change across tasks or across time, and if the -tatstatiyn of the intrinsic reward affects its overall scale.
248
249
Diversity for diversity's sake One role of a general-purpose exploration method to allow an agent to get off the ground in wide variety of domains. While this may clash with sample-efficient
8

---

251
learning on specific domains, we believe that the former objective will come to dominate in the long run. In this light, methods that exhibit more diverse behaviour are preferable for that reason alone, because they are more likely to escape local optima or misaligned priors.
252
254
Related work While not the most common approach to exploration in RL, we are aware of some notable work that has investigated non-trivial temporal structure. The cz-greedy algorithm [Dabney et al.. 2020] is inspired by Levy flights in nature [Baronchelli and Radicchi, 2013] and initiates contiguous chunks of directed behaviour ('flights") with the length sampled from heavy-tailed distribution. In contrast to our proposal, these flights act with a single constant action, instead of invoking an explore mode. [Campos et al., 2021] pursue a similar idea, but with flights along pre-trained trained coverage policies, while [Ecoffet et al., 2021] chain a 'returnto--sttte policy to an explore mode. Maybe closest to our X' setting is [Bagot al., 2020], where periods of intrinsic reward pursuit are explicitly invoked by the agent. Exploration with gradual change instead of abrupt mode switches, appears generally at long time-scales, such when pursuing intrinsic rewards [Schmidhuber, 2010, Oudeyer and Kaplan, 2009], but can also be effective at shorter scales e.g., Never-GGve-Ll [Badia et al., 2020b]. Related work on the question of which states to prefer for exploratory decisions Tookic 2010] tends to not consider starting prolonged exploratory periods.
255
256
258
280
262
263
264
285
286
Relation to options Ideas related to switching behaviours at intra-episodic time-scales well- known outside of the context of exploration, the best-known framework being options in hierarchical RL, where the goal is to chain together sequence of sub-behaviours into a reward-maximist policy [Sutton et al., 1999, Mankowitz et al., 2016]; but some work has looked at using options for exploration too [Jinnai et al.. 2019a, Bougie and Ichise, 2021]. In its full generality, the options framework is a substantially more ambitious endeavour than our proposal, as it requires learning a full state-dependent hierarchical policy that picks which option to start (and when), as well as jointly learning the options themselves.
268
269
270
271
272
273
274
275 L Limitations
Our proposed approach inherits many of the challenges that are typical for exploration methods, such as sample efficiency or trading off risk. An aspect that is particular to the intra-episode switching case the different nature of the off aaf-policy-ness. The resulting effective policy can produce state distributions that differ substantially from those of either of the two base mode behaviours that are being interleaved. can potentially visit parts of the state space that neither base policy would reach if followed from the beginning of the episode. While boon for exploration, this might pose a challenge to learning, as could require off- fff-policy corrections that treat those states differently and do not only correct for differences in action space. We leave this as an intriguing consideration for future work; this paper does not use any non non-trivial ool-policy correction (see Appendix A).
276
277
278
279
281
282
283
Future work With the dimensions laid out in Section 2, it should be clear that this paper can but scratch the surface. We see numerous opportunities for future work, on some of which we already carried out initial investigations, see Appendix B. For starters, there is no inherent need to restrict the mechanism to just two modes: A richer form of exploration could switch between exploit, explore, novelty and mastery [Thomaz and Breazeal, 2008], between many diverse forms of exploration (such as different levels of optimism [Derman et al., 2020, Moskovitz et al., 2021J). It is also conceivable to switch less abruptly; for example, if both exploit- and explore-mode mode behaviours are induced by a reward function, a Q-value-basse
285
287
289
291
agent with successor features [Barreto et al., 2017, Borsa et al., 2019] could interpolate between them to make switching more gradual [Barreto et al., 2019]. Triggers are another aspect that could be expanded or refined: there are different candidates
292
293
294
for estimating uncertainty, such as ensemble discrepancy [Wiering and Van Hasselt, 2008, Buckman
285
et al., 2018], amortised value errors [Flennerhag et al., 2020], or density models [Bellemare et al., 2016, Ostrovski et al., 2017]; also, triggers could be based on other signals that not derived from
296
297
uncertainty, such as salience [Downar et al., 2002], minimal coverage [Jinnai et al., 2019a,b), or empowerment [Klyubin et al., 2005, Gregor et al., 2016, Houthooft et al., 2016].
298
299
We have presented an initial study of intra-episodic explooooiiiiiieelt centred on the scenario of switching between an explore and an exploit mode. We hope this has broadened the available forms of temporal structure in behaviour, leading to more diverse, adaptive and intentional forms of exploration, in turn enabling RL to scale ever more complex domains.
Conclusion
300
301
302

---

References
303
304
M. A. Addicott, J. M. Pearson, M. M. Sweitzer, D. L.. Barack, and M. L. Platt. A Primer on Foraging and the Explore/Exploit Trade-Off for Psychiatry Research. Nenooyychpphamaaclgg: 42(10): 1931-1939, Sep 2017.
306
A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, D. Guo, and C. Blundell. Agent57: Outperforming the Atari human benchmark, 2020a.
308
A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky,
310
A. Pritzel, A. Bolt, and C. Blundell. Never give up: Learning directed exploration strategies, 2020b.
312
L. Bagot, K. Mets, and S. Latré. Learning intrinsically motivated options stimulate policy exploration, 2020.
314
A. Baronchelli and Radicchi. Lévy flights in human behavior and cognition. Chaos, Solitons & Fractals, 56:::::::100 105-10,, 2013.
315
316 A. Barreto, W. Dabney, R. Munos, J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, pages 4055 4065, 2017.
317
318
319 A. Barreto, D. Borsa, S. Hou, G. Comanici, E. Aygün, P. Hamel, D. Toyama, J. hunt, S. Mourad, D. Silver, and D. Precup. The option keyboard: Combining skills in reinforcement learning. In Advances in Neural Information Processing Systems 32, pages 13052-13062, 2019.
320
321
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligncee Research, 47:253-279,
323
324
2013.
325
M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Neural Information Processing Systems, 2016.
327
D. Borsa, A. Barreto, Quan, D. J. Mankowitz, H. van Hasselt, R. Munos, D. Silver, and T. Schaul. Universal successor features approximators. In International Conference on Learning Representa- tions, 2019.
329
N. Bougie and R. Ichise. Fast and slow curiosity for high-level exploration in reinforcement learning. Applied Intelligence, 51(2): 1087 2021.
331
332
J. Bradbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPlas, Wanderman- Miine, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http::::/ggttttt.oc wtt.ggt....oggoggggg.....o.o.. com/googee/aaaa
333
335 J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. arXiv preprint I 2018.
336
337 D. Budden, M. Hessel, Kemaev, Spencer, and F. Viola. Chex: Testing made fun, in jax!, 2020a. URL http::/gitee thub.b..ttttttgggggg com/ http//deemmncheeeeee chex.
338
D. Budden, M. Hessel, J. Quan, S. Kapturowski, K. Baumli, S. Bhupatiraju, A. Guy, and M. King. RLax: Reinforcement Learning in JAX, 2020b. URL htttp://githul com htmm/deeeminddlee.
340
Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov. Exploration by random network distillation. CoRR, abs/1 fas/1810.. 12894, 2018
342
V. Campos, P. Sprechmann, S. Hansen, A. Barreto, S. Kapturowski, A. Vitvitskyi, A. P. Badia, and C. Blundell. Coverage as a principle for discovering transferable behavior in reinforcement learning, arXiv preprint 1322v:2102.13515, 2021.
344
346
O. Cappé, A. Garivier, O.-A. Maillard, R. Munos, G. Stoltz, al. Kullhack-leibler upper confidence bounds for optimal sequential allocation. Annals of Statistics, 1163)1151--1441, 2013, 2013.
10

---

348
A. Cassirer, G. Marth,Maron, T. Sottiaux, M. Kroiss, and E. Brevdo. Reverb: An efficient data storage and transport system for ml research, 2020. URL https://gittuu. ctt htmm//deemndd/eeeee.e
349
350
Cathomas, F, Klaus, K. Guetter, H.-K. Chung, A. Raja Beharelle, T. R. Spiller, R. Schlegel, E. Seifritz, M. N. Hartmann-Riemer, P. N. Tobler, and S. Kaiser. Increased random exploration in schizophrenia associated with inflammation. npj Schizophrenia, 7(1);6, Feb 2021.
352
353
K. Chakroun, D. Mathar, A. Wiehler, F. Ganzer, and J. Peters. Dopaminergic modulation of the exploratiom/explootttonn trade-off in human decision-making. Elife, 9, 06 2020.
D. Churchill and M. Buro. Build order optimization in starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 6(1), 2011.
356
357
J. D. Cohen, S. M. McClure, and A. J. Yu. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1481 ):333 ):933-942, 2007.
358
359
V. D. Costa, A. R. Mitz, and B. B. Averbeck. Subcortical substrates of explore-exploit decisions in primates. Neuron, 103())533-545.e55 Aug. 2019.
361
362
J. Cremer, T. Honda, Y. Tang, J. Wong-Ng, M. Vergassola, and T. Hwa. Chemotaxis as a navigation strategy to boost range expansion. Nature, 575(7784):658-663) 11 2019.
W. Dabney, G. Ostrovski, and A. Barreto. Temporally-exteedded c-greedy exploration, 2020.
365
E. Derman, D. Mankowitz, T. Mann, and S. Mannor. A bayesian approach robust reinforcement learning. In Uncertainty in Artificial Intelligence, pages 648-658. PMLR, 2020.
386
387 J. Downar, A. P. Crawley, J. Mikulis, and K. D. Davis. A cortical network sensitive to stimulus salience in a neutral behavioral context across multiple sensory modalities. Journal of neurophysi- ology, 87(1):615-622 2002.
368
389
370
R. B. Ebitz, B. J. Sleezer, H. P. Jedema, C. W. Bradberry, and B. Y. Hayden. Tonic exploration governs both flexibility and lapses. PLOS Computational Biology, 15(11):e1007475 Nov. 2019.
371
372 A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. Go-explore: a new approach for hard-exploration problems, 2021.
S. Flennerhag, J. X. Wang, P. Sprechmann, F. Visin, A. Galashov, S. Kapturowski, D. L. Borsa, N. Heess, A. Barreto, and R. Pascanu. Temporal difference uncertainties as a signal for exploration. arXiv preprint arXiv:20/0.02255, 2020.
375
376
377 S. J. Gershman. Deconstructing the human algorithms for exploration. Cognition, 173:34 2018. ISSN 0010-0277.
378
379
S. J. Gershman and B. G. Tzovaras. Dopaminergic genes are associated with both directed and random exploration. bioRxiv, 2018.
380
381
K. Gregor, J. Rezende, and Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507, 2016.
T. Hennigan, T. Cai. T. Norman, and I. Babuschkin. Haiku: Sonnet for JAX, 2020. URL http: //github. com/ www/ http//deemmnddd/e /m/ haiku.
384
385
M. Hessel, D. Budden, F. Viola, M. Rosca, E. Sezener, and T. Hennigan. Optax: Composable gradient
386
transformation and optimisation, in JAX!, 2020. URL http:://gtth/: com/ /optt/xx.......//////oo///
387
T. T. Hills, P. M. Todd, D. Lazer, A. D. Redish, and I. D. Couzin. Exploration versus exploitation in space, mind, and society. Trends Cogn Sci, 19)444,,,, Jan 2015.
388
389
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98):: 1997.
390
381
D. Horgan, J. Quan, Budden, G. Barth-Maron, M. Hessel, H. Van Hasselt, and D. Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018.
11

---

393
R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. Vime: Variational information maximizing exploration. arXiv preprint 1605.0660.0900 2016.
394
395
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv: 1XX.::661..0597,, 2016.
T. Jaksch, R. Ortner, and P. Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11(4), 2010.
398
389
Y. Jinnai, J. W. Park, D. Abel, and G. Konidaris. Discovering options for exploration by minimizing cover time. In International Conference Machine Learning, pages 3130-3139. PMLR, 2019a.
Y. Jinnai, J. W. Park, M. C. Machado, and G. Konidaris. Exploration in reinforcement learning with deep covering options. In International Conference on Learning Representations, 2019b.
402
403
S. Kapturowski, G. Ostrovski, W. Dabney, J. Quan, and R. Munos. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2019.
404
J. M. Kembro, M. Lihoreau, J. Garriga, E. P. Raposo, and F. Bartumeus. Bumblebees learn foraging routes through exploiaaiion-explrrtiin cycles. R Soc Interface, 16(156):20190103, 07 2019.
407
408
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 111::9442.69800, 2014.
A. S. Klyubin, D. Polani, and C. L. Nehaniv. Empowerment: A universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume I, pages 128-135. IEEE, 2005.
411
412
413
Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
414
C. Linke, M. Ady, M. White, T. Degris, and A. White. Adapting behaviour via intrinsic reward: survey and empirical study, 2019.
415
M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
417
D. J. Mankowitz, T. A. Mann, and S. Mannor. Adaptive skills adaptive partitions (ASAP). In Neural Information Processing Systems, 2016.
420
421
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforce- ment learning. Nature, 518(7540):529-533 2015.
422
423
424
425
T. Moskovitz, J. Holder, A. Pacchiano, and M. Arbel. Deep reinforcement learning with dynamic optimism. arXiv preprint arXiv:2 2021.
426
G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos. Count-based exploration with neural density models. CoRR, abs/ 703.01310, 2017.
428
429
P. Y. Oudeyer and Kaplan. What intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
T. G. Power. Play and exploration in children and animals. Psychology Press, 1999.
432
T. Schaul, J. Quan, Antonoglou, and D. Silver. Prioritized experience replay. In International Conference on Learning Representations, Puerto Rico, 2016.
433
434
T. Schaul, D. Borsa, D. Ding, D. Szepesvari, G. Ostrovski, W. Dabney, and S. Osindero. Adapting behaviour for learning progress, 2019.
435
436
Schaul, G. Ostrovski, I. Kemaev, and D. Borsa. Return- based sealing: Yet another normalisation trick for deep RL. arXiv preprint arXiv:2105.05347 2021.
12

---

438
J. Schmidhuber. Curious model-building control systems. In Proc. international joint conference on neural networks, pages 1458-1463, 1991.
439
440
Schmidhuber. Formal theory creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions Autonomous Mental Development, 2(3):230-247, 2010.
E. Schulz, R. Bhui, B. C. Love, B. Brier, M. T. Todd, and S. J. Gershman. Structured, uncertainty- driven exploration real-world consumer choice. Proceedings of the National Academy of Sciences, (116(2)::3393-111
443
June 2019.
445
R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
448
R. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 12(1-2)1181-2111 1999.
447
A. L. Thomaz and C. Breazeal. Experiments in socially guided exploration: Lessons learned in building robots that learn with and without human teachers. Connection Science, 20(2-3):91-100, 2008.
449
S. B. Thrun. Efficient exploration in reinforcement learning, 1992.
M. Tokic. Adaptive s-greedy exploration in reinforcement learning based value differences. In Annual Conference on Artificial Intelligence, pages 203-210. Springer, 2010.
453
454
G. G. Turrigiano and S. B. Nelson. Homeostatic plasticity in the developing nervous system. Nature reviews neuroscience, 5(2):97-107, 2004.
455
456
A. Waltz, R. C. Wilson, A. Albrecht, M. J. Frank, and M. Gold, Differential effects of psychotic illness on directed and random exploration. Compulational Psychiatry, 4(0):18, Aug. 2020.
458
459
Z. Wang. T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1995-2003, New York, New York, USA, 20-22 Jun 2016. PMLR.
461
M. A. Wiering and H. Van Hasselt. Ensemble algorithms in reinforcement learning. IEEE Transac- tions on Systems, Man, and Cybernetics, Part B (Cybernetics), 38(4):930 936, 2008.
465 J. M. Wolfe, K. R. Cave, and S. L. Franzel. Guided search: an alternative to the feature integration model for visual search. Exp Psychol Hum Percept Perform, 15(3):419 433, 1989.
486
467
D. Zha, W. Ma, L. Yuan, X. Hu, and J. Liu, Rank the episodes: A simple approach for exploration in procedurally-generaae environments, In International Conference Learning Representations, 2021.
489
13

---

Checklist
470
471
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
473
contributions and scope? [Yes]
474
(b) Did you descrihe the limitations of your work? [Yes] See Section 4
475
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
477
478
2. If you are including theoretical results...
479
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
480
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (cither in the supplemental material or as a URL)? [No] We did not include code, but described the specifics of our methods in sufficient detail to reproduce results.
483
485
486
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.
487
488
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Appendix A.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.
491
492
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
493
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite all open- source libraries used, see Appendix A.
494
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
497
498
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
499
500
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
502
5. If you used crowdsourcing or conducted research with human subjects...
503
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
505
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
508
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
508
14


Reviews:
[{"review_id": "huVV1l14Vn", "reviewer": "Reviewer_SAR7", "summary": "This paper proposes several key parameters for studying when to switch between exploration and exploitation modes in hard-exploration RL problems. These parameters include frequency and timescale for exploration mode, how the mode is triggered, and how exploration is performed. The authors show that in certain circumstances, correctly setting these parameters can lead to significant gains in task performance, and discuss methods for setting these parameters without significant hyperparameter tuning. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall. Learning when to explore is an important problem for meta-learning or continual learning. The parameters explored in the paper are presented well, although it would be helpful to show more random seeds. The paper discusses many interesting properties of intra-episode exploration, but ultimately the effect of these properties on task performance appears inconclusive. \n\nPros:\n1) The paper was well organized and the writing was clear. \n2) Experimental parameters are clearly defined, and ablations are thorough. \n3) There are some surprising results, such as the importance of starting mode and of step based vs probabilistic switching. \n\nCons:\n1) While learning when to explore is an interesting problem, in the setting of the paper (Hard-exploration Atari games with up to 1B frames' data) there are many other methods that are relevant and comparable [1, 2, 3]. In particular, it would be interesting to see comparisons to a method that simultaneously uses intrinsic/exploration and extrinsic rewards. \n2) I would be interested in seeing more seeds per experiment. \n3) The paper focuses on the potential benefits on intra-episode mode-switching, but it doesn't appear that this method significantly outperforms episode-based switching overall. \n\nQuestions: \n1) In Figure 3, why are the XI-experiment-level-G curves different between the left two and right two plots? Does exploration mode affect the greedy policies?\n2) Do you have an intuition about why the blind trigger may outperform the intrinsic one, as in Phoenix, or when uniform exploration may outperform intrinsic motivation?\n3) Could you provide some further discussion about how option-based exploration relates to this work? \n\nDetails:\n- (Section 3.1 Line 169) - end -> and\n- (Figure 5 and 6) - Are different seeds used for these experiments? \n\n[1] Ecoffet, A., Huizinga, J., Lehman, J. et al. First return, then explore. Nature 590, 580–586 (2021). https://doi.org/10.1038/s41586-020-03157-9\n\n[2] Y. Burda, H. Edwards, A. Storkey, O. Klimov. Exploration by Random Network Distillation.\n\n[3] J. Martin, S Narayanan S., T Everitt, M Hutter. Count-Based Exploration in Feature Space for Reinforcement Learning\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address the limitations of this work, but it would be helpful to include a discussion about its social impacts. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "SCKtr65rdsh", "reviewer": "Reviewer_h8V7", "summary": "The paper focuses on switching between exploration and exploitation modes in RL. Specifically, it examines the temporal granularity of exploration periods, decisions of when to switch between explore and exploit modes, and comparisons between different types of exploration. The work uses  a ‘value promise discrepancy’ for informed switching between modes. Experimental results are presented on a number of Atari games. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The work focuses on the question of when to switch between different modes. This temporal aspect of exploration is frequently overlooked in prior work on exploration, which tends to focus more on the type of exploration. This temporal analysis is interesting and sheds more light on interpreting RL.\n\nQuality: The work is technically sound, and the experimental results are detailed with many different visualizations. The Atari environments used are quite challenging, 5 of which are hard exploration games that prior work frequently struggles with or does not address.\n\nClarity: The work is generally well-written. I do think some parts of the paper could be reorganized for improved clarity. For example, sections 2.3 and 2.4 both cover switching and could be combined. Section 3.1 is very helpful for understanding the variants used. Some of the results could be better explained: figures 4 and 6 took quite a bit of time to understand, as these are visualization techniques unique to this paper. Other parts could have better analysis (see figure 3 and figure 5 notes below). \n\nSignificance: The paper brings temporal analysis to the forefront of RL, which could be interesting to many researchers. The impact could be greater with clearer take-aways. The overhead of using a metacontroller with many parameters to tune will make it challenging for researchers to incorporate the ideas. Clear take-aways (e.g. which fixed exploration strategy is best) that can be used without the entire meta-controller setup would allow many people to make simple tweaks to improve their RL work.\n\nMinor comments:\nLine 93 states that blind switching “does not take state or time into account” - but blind switching *only* depends on time.\n\nFigure 3: I don’t understand why the experiment-level uniform exploration performance improves over time - shouldn’t this be a random policy, and therefore have flat reward curves? From the extension in the appendix (figure 9), it seems that Gravitar and Phoenix are the most increasing curves. Phoenix is especially surprising, where the pure exploration curve ends up with higher performance than all other methods. I’m also surprised that the experiment-level uniform exploration looks to be similar performance as experiment-level RND, and would like to see discussion of this. The large variance on these aggregate plots also makes it difficult to draw conclusions from (more than 3 seeds could help with this). I’d like to see analysis on why intra-episodic exploration performs worse than baselines on Montezuma’s revenge in both cases (appendix Figure 9).\n\nFigure 5: The illustration of temporal structure is really interesting. I’d like to see more analysis on this, for example what scenarios happen during the dense periods of exploration.\n\nFigure 6: I’d be curious to see the analogous results for non-blind runs. Also, could the meta-controller decide whether to start in explore or exploit mode? Given that which is better is heavily game-dependent, this seems like something well-suited to a meta-controller.\n\nFigure 7: A colorbar legend would be helpful here, especially for readers who are colorblind or reading a black and white printed copy.\n\nRelated work: This section is quite short. I’d like to see more situating of this paper relative to prior work on non-temporal exploration. A section on bonus-based exploration would be helpful.\n\nAdditional methods: I’d like to see a comparison to ε decay (instead of fixed ε in the ε-greedy baseline), since this is commonly used in RL work. I’d also like to see the intrinsic results where exploit mode does not include the auxiliary task (modes are entirely separate), as this might be a better comparison to the uniform exploration cases.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I’d like to see more discussion of limitations. There are many variants of the method and parameters to tune, so the complexity of this approach might make use in other domains difficult.\n\nCompute needs: there is no mention of the substantial compute required in the paper, and the section in Appendix A could be elaborated on. 2 Billion frames per run * 3 seeds * 5 methods * 2 settings = 60 billion frames, not including any experimentation or hyperparameter tuning. In addition to the environmental impact, this level of computation is not accessible to most researchers. Limitations of compute needs and lack of sample efficiency should be addressed.\n\nThe checklist says N/A for societal impacts; even if the authors don’t see the need to include a section in the paper, I’d like to see a justification of why this is the case.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "QEmT-DBQlj", "reviewer": "Reviewer_XEyz", "summary": "The paper argues that for effective exploration, switching behavior modes is important. Specifically, the method proposed switches behavior from exploit to explore for chunks within an episode. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Significance** : While it is true that exploration is an important problem, and also that monolithic exploration is not a great approach, it is unclear if we really need sub-episodic switching. This is since switching exploratory behavior at the episodic level already works quite well. In fact on the Atari environments evaluated, episodic switching actually performs much better that the proposed approach on Montezuma's revenge (from training curves in the appendix, as well as the bar plot in figure 4), and is not significantly worse on the other Atari games. As such the main emphasis of the paper seem to be on the important of switching behavior within an episode, and I don't think the result show it to be that much more important that episodic switching. \n\n**Originality** : The idea for the exploration trigger to be based on TD error is not very novel and has been studied in prior work [1]. Furthermore this is similar in spirit to prior work where exploration is done in regions of epistemic uncertainty. The paper does discuss related work in related to exploration with 'non-trivial temporal structure' as well as some work related to options, but this is quite brief, and doesn't include a comprehensive list of works for exploration in RL. \n\n**Clarity, quality**: The ideas in the paper are well presented and easy to follow. While the authors test on Atari, they don't include any evaluation on continuous control environments, such as simulated robot environments, and so it's unclear if the insights are applicable in that domain. The main issue remains that it's unclear if sub-episodic exploration helps performance, given the results in the paper. \n\n[1] : Reward Prediction Error as an Exploration Objective in Deep RL (IJCAI 20)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations discussed above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "93t1-9tgmsG", "reviewer": "Reviewer_v6F4", "summary": "This paper studies the question of whether there is value in switching between exploration and exploitation behavior in RL. The authors investigate different signals for switching between learning a policy which maximizes an intrinsic reward (RND) and exploiting an extrinsic reward. The authors evaluate their approach on a number of Atari games, and show that an intra-episodic exploration scheme that switches based upon a \"value discrepancy\" is comparable/better to \"episode\" level exploration scheme on some evaluations. The authors additionally contribute some analysis showing the qualitative switching behavior and its effect on downstream performance.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n\nThis paper investigates a setting that is not commonly studied in the RL literature (namely one where exploration and exploitation are explicitly decoupled). While I did not find the particular \"value promise\" trigger to be well motivated in the writing, to my knowledge such a scheme is novel to the best of my knowledge. \n\nQuality\n\nOverall, I feel that this paper provides some good exploratory experiments which suggests that there is some merit to this idea. However, I think there are some high level conceptual/experimental issues which I feel need to be resolved before publication. \n\nThere are some clear conceptual issues with how this paper explicitly decouples the exploration and exploitation schemes in RL from an optimization perspective. The main one being that when done with a novelty seeking bonus such as RND, the data produced by exploration to be very off-policy from the current exploitation policy. This causes me to be skeptical of the overall stability/scalability of this type of approach, even with the addition of off-policy correction terms. From a practical perspective, this paper also introduces a layer of complexity in determining the switching procedure between this behavior mode. The authors attempt to address this with a bandit based  controller, but given somewhat incremental improvements in performance I do not feel this additional complexity is justified. \n\nFrom an experimental perspective, another concern I have is that this paper does not compare to other approaches such as Bagot et al., 2020 which also proposes a similar high level scheme. While this paper is cited in the related work (among other papers), the authors neither provide a comparison nor give a reason why a comparison is not appropriate.  \n\nClarity\n\nOverall, I found the exposition in this paper to be somewhat opaque. I was not able to figure out important experimental details from reading the main paper (e.g., the underlying RL algorithm). I also found the captions and plots a little difficult to parse (e.g., Fig 3, all of the plots share the same title). Finally I would appreciate a dedicated related work section beyond the single paragraph provided. I feel it is the author's duty to situate their contribution in the literature clearly, but I felt in the current writing this is distributed throughout the paper in an unnecessary way. \n\nSignificance\n\nI feel that this paper starts with an interesting premise, but falls short in delivering clear evidence that 1) the exploration scheme proposed is clearly superior to conventional exploration schemes, 2) such an approach would not result in the need for significant off-policy correction, 3) the method proposed here is better than other similar schemes proposed previously. Due to this, I do not believe this paper in its current form is ready for publication without significant improvements. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors also raise the concern of the problem of \"off-policy-ness\" in their limitation section. While I feel they understate the difficulties of addressing this issue, I feel that their self-evaluation of their limitations is reasonable. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on an exploration question the authors consider under-explored: "when to explore". Whereas some other exploration methods are either 'monolithic' (term used in the paper for strategies where the action selection at each timestep has both exploration and exploitation characteristics), or decides to use a purely explorative or exploitative behavior for either a single time step or a whole episodes, the authors propose a strategy for switching within an episode to stretches of an 'exploit' mode and an 'explore' mode. 

The review process did not have a lot of interaction between reviewers and authors, such that I have taken a closer look at the paper itself and at the replies of the authors to determine which of the points raised by the reviewers have been sufficiently addressed. 

On originality, the reviewers mostly agreed that the question of 'when' to explore was a new perspective, though one reviewer didn't find the approach itself wholly original. 

On technical quality, there were several comments raised by the reviewers. One is that the experimental environments are narrowly chosen: all experiments are on Atari games, which have strong commonality on certain characteristics (e.g., being (close to) deterministic). Another is that potential benchmarks are missing: although some of the benchmarks are used a lot in practice, a strategy like epsilon-greedy is not a particularly strong baseline. Instead, a 'monolithic' exploration method could have been used (e.g. a combination of intrinsic and extrinsic rewards as suggested by a reviewer, or perhaps posterior sampling). 
Moreover, I found the motivation for both mode-switching based exploration and the particular implementation (e.g. the proposal for 'value promise trigger') on the weak side. Many 'monolithic' methods (e.g. posterior sampling) would implicitly also switch between more 'exploration focused' or 'exploitation focused' behavior, why is switching between 'pure' exploration or exploitation important? The value promise trigger would I think trigger with either a non-converged value function (appropriately) or when the environment is intrinsically very random (where more exploration might not be sensible). I think the Atari environments are not a good test-bed as they are (almost) deterministic. Furthermore, at the moment the value trigger is high, the problematic state might be k steps ago. 

On significance, most reviewers considered the comparative results between episode-level switching and the proposed strategy unconvincing. Here, the authors argued that (1) on all games, some intra-episode switching does perform best (2) there is some improvement overall on all games and (3) even when performance is not significantly better, intra-episode switching is still interesting. There is some merit to all of these argument, though the improvement is small overall and the variance is high (also because a relative low number of seeds is used). I find the third argument most interesting: in continuing tasks, one doesn't have the 'luxury' to re-set and thus intra-episode switching would be a good alternative to the now-impossible episode-level switching. 

The writing was mostly considered clear by the reviewers though there were some concrete comments for improvement. Some confusion about the meaning of Fig 3 was not completely resolved by the authors reply. 

All in all, I think that although the paper does present a new viewpoint, it is currently not clear enough how significant this contribution is in the state of the field, as experimental evidence is somewhat inconclusive and the motivation of design decisions is unclear. 

Minor issue: 
I think there are a number of issues with the use of gamma around line 120. States farther in the future from t-k are less this counted than closeby states, and the value at t isn't discounted at all. 
Decision: Reject