When should agents explore?

Exploration remains a central challenge for reinforcement learning ( RL ) . Virtually all existing methods share the feature of a monolithic behaviour policy that changes only gradually ( at best ) . In contrast , the exploratory behaviours of animals and humans exhibit a rich diversity , namely including forms of switching between modes . This paper presents an initial study of mode-switching , nonmonolithic exploration for RL . We investigate different modes to switch between , at what timescales it makes sense to switch , and what signals make for good switching triggers . We also propose practical algorithmic components that make the switching mechanism adaptive and robust , which enables flexibility without an accompanying hyper-parameter-tuning burden . Finally , we report a promising and detailed analysis on Atari , using two-mode exploration and switching at sub-episodic time-scales . 1 INTRODUCTION . The trade-off between exploration and exploitation is described as the crux of learning and behaviour across many domains , not just reinforcement learning ( Sutton & Barto , 2018 ) , but also in decision making ( Cohen et al. , 2007 ) , evolutionary biology ( Cremer et al. , 2019 ) , ecology ( Kembro et al. , 2019 ) , neuroscience ( e.g. , focused versus diffuse search in visual attention ( Wolfe et al. , 1989 ) , dopamine regulations ( Chakroun et al. , 2020 ) ) , cognitive sciences ( Hills et al. , 2015 ) , as well as psychology and psychiatry ( Addicott et al. , 2017 ) . In a nutshell , exploration is about the balance between taking the familiar choice that is known to be rewarding and learning about unfamiliar options of uncertain reward , but which could ultimately be more valuable than the familiar options . Ample literature has studied the question of how much to explore , that is how to set the overall trade-off ( and how to adjust it over the course of learning ) ( Jaksch et al. , 2010 ; Cappé et al. , 2013 ; Lattimore & Szepesvári , 2020 ; Thrun , 1992 ) , and the question of how to explore , namely how to choose exploratory actions ( e.g. , randomly , optimistically , intrinsically motivated , or otherwise ) ( Schmidhuber , 1991 ; Oudeyer & Kaplan , 2009 ; Linke et al. , 2019 ) . In contrast , the question of when to explore has been studied very little , possibly because it does not arise in bandit problems , where a lot of exploration methods are rooted . The ‘ when ’ question and its multiple facets are the subjects of this paper . We believe that addressing it could lead to more intentional forms of exploration . Consider an agent that has access to two modes of behaviour , an ‘ explore ’ mode and an ‘ exploit ’ mode ( e.g. , a random policy and a greedy policy , as in ε-greedy ) . Even when assuming that the overall proportion of exploratory steps is fixed , the agent still has multiple degrees of freedom : it can explore more at the beginning of training and less in later phases ; it may take single exploratory steps or execute prolonged periods of exploration ; it may prefer exploratory steps early or late within an episode ; and it could trigger the onset ( or end ) of an exploratory period based on various criteria . Animals and humans exhibit non-trivial behaviour in all of these dimensions , presumably encoding useful inductive biases that way ( Power , 1999 ) . Humans make use of multiple effective strategies , such as selectively exploring options with high uncertainty ( a form of directed , or informationseeking exploration ) , and increasing the randomness of their choices when they are more uncertain ( Gershman , 2018 ; Gershman & Tzovaras , 2018 ; Ebitz et al. , 2019 ) . Monkeys use directed exploration to manage explore-exploit trade-offs , and these signals are coded in motivational brain regions ( Costa et al. , 2019 ) . Patients with schizophrenia register changes in directed exploration and experience low-grade inflammation when shifting from exploitation to random exploration ( Waltz et al. , 2020 ; Cathomas et al. , 2021 ) . This diversity is what motivates us to study which of these can benefit RL agents in turn , by expanding the class of exploratory behaviours beyond the commonly used monolithic ones ( where modes are merged homogeneously in time ) . 2 METHODS . The objective of an RL agent is to learn a policy that maximises external reward . At the high level , it achieves this by interleaving two processes : generating new experience by interacting with the environment using a behaviour policy ( exploration ) and updating its policy using this experience ( learning ) . As RL is applied to increasingly ambitious tasks , the challenge for exploration becomes to keep producing diverse experience , because if something has not been encountered , it can not be learned . Our central argument is therefore simple : a monolithic , time-homogeneous behaviour policy is strictly less diverse than a heterogeneous mode-switching one , and the former may hamstring the agent ’ s performance . As an illustrative example , consider a human learning how to ride a bike ( explore ) , while maintaining their usual happiness through food , sleep , work ( exploit ) : there is a stark contrast between a monolithic , time-homogeneous behaviour that interleaves a twist of the handlebar or a turn of a pedal once every few minutes or so , and the mode-switching behaviour that dedicates prolonged periods of time exclusively to acquiring the new skill of cycling . While the choice of behaviour in pure exploit mode is straightforward , namely the greedy pursuit of external reward ( or best guess thereof ) , denoted by G , there are numerous viable choices for behaviour in a pure explore mode ( denoted by X ) . In this paper we consider two standard ones : XU , the naive uniform random policy , and XI , an intrinsically motivated behaviour that exclusively pursues a novelty measure based on random network distillation ( RND , ( Burda et al. , 2018 ) ) . See Section 4 and Appendix B for additional possibilities of X . In this paper we choose fixed behaviours for these modes , and focus solely on the question of when to switch between them . In our setting , overall proportion of exploratory steps ( the how much ) , denoted by pX , is not directly controlled but derives from the when . 2.1 GRANULARITY . An exploration period is an uninterrupted sequence of steps in explore mode . We consider four choices of temporal granularity for exploratory periods , also illustrated in Figure 1 . Step-level exploration is the most common scenario , where the decision to explore is taken independently at each step , affecting one action.1 The canonical example is ε-greedy ( Fig.1 : C ) . Experiment-level exploration is the other extreme , where all behaviour during training is produced in explore mode , and learning is off-policy ( the greedy policy is only used for evaluation ) . This scenario is also very common , with most forms of intrinsic motivation falling into this category , namely pursuing reward with an intrinsic bonus throughout training ( Fig.1 : A ) .2 Episode-level exploration is the case where the mode is fixed for an entire episode at a time ( e.g. , training games versus tournament matches in a sport ) , see Fig.1 : B . This has been investigated for simple cases , where the policy ’ s level of stochasticity is sampled at the beginning of each episode ( Horgan et al. , 2018 ; Kapturowski et al. , 2019 ; Zha et al. , 2021 ) . Intra-episodic exploration is what falls in-between step- and episode-level exploration , where exploration periods last for multiple steps , but less than a full episode . This is the least commonly studied scenario , and will form the bulk of our investigations ( Fig.1 : D , E , F , G ) . We denote the length of an exploratory period by nX ( and similarly nG for exploit mode ) . To characterise granularity , our summary statistic of choice is medX : = median ( nX ) . Note that there are two possible units for these statistics : the raw steps or the proportion of the episode length L. The latter has different ( relative ) semantics , but may be more appropriate when episode lengths vary widely across training . We denote it as rmedX : = median ( nX /L ) . 1The length of an exploratory period tends to be short , but it can be greater than 1 , as multiple consecutive step-wise decisions to explore can create longer periods . 2Note that it is also possible to interpret ε-greedy as experiment-level exploration , where the X policy is fixed to a noisy version of G . 2.2 SWITCHING MECHANISMS . Granularity is but the coarsest facet of the ‘ when ’ question , but the more precise intra-episode timings matter too , namely when exactly to start and when to stop an exploratory period . This section introduces two mechanisms , blind and informed switching . It is worth highlighting that , in general , the mechanism ( or its time resolution ) for entering explore mode differs from the one for exiting it ( to enter exploit mode ) – this asymmetry is crucial to obtain flexible overall amounts of exploration . If switching were symmetric , the proportion would be pX ≈ 0.5 . Blind switching The simplest switching mechanism does not take any state into account ( thus , we call it blind ) and is only concerned with producing switches at some desired time resolution . It can be implemented deterministically through a counter ( e.g. , enter explore mode after 100 exploit mode steps ) , or probabilistically ( e.g. , at each step , enter explore mode with probability 0.01 ) . Its expected duration can be parameterised in terms of raw steps , or in terms of fractional episode length . The opposite of blind switching is informed switching , as discussed below . Informed switching Going beyond blind switching opens up another rich set of design choices , with switching informed by the agent ’ s internal state . There are two parts : first , a scalar trigger signal is produced by the agent at each step , using its current information – drawing inspiration from human behaviour , we view the triggering signal as a proxy for uncertainty ( Schulz et al. , 2019 ) : when uncertainty is high , the agent will switch to explore . Second , a binary switching decision is taken based on the trigger signal ( for example , by comparing it to a threshold ) . Again , triggering will generally not be symmetric between entering and exiting an exploratory period . To keep this paper focused , we will look at one such informed trigger , dubbed ‘ value promise discrepancy ’ ( see Appendix B for additional competitive variants ) . This is an online proxy of how much of the reward that the agent ’ s past value estimate promised ( k steps ago ) have actually come about . The intuition is that in uncertain parts of state space , this discrepancy will generally be larger than when everything goes as expected . Formally , Dpromise ( t− k , t ) : = ∣∣∣∣∣V ( st−k ) − k−1∑ i=0 γiRt−i − γkV ( st ) ∣∣∣∣∣ where V ( s ) is the agent ’ s value estimate at state s , R is the reward , and γ is a discount factor . Starting mode When periods last for a significant fraction of episode length , it also matters how the sequence is initialised , i.e. , whether an episode starts in explore or in exploit mode , or more generally , whether the agent explores more early in an episode or more later on . It is conceivable that the best choice among these is domain dependent ( see Figure 6 ) : in most scenarios , the states at the beginning of an episode have been visited many times , thus starting with exploit mode can be beneficial ; in other domains however , early actions may disproportionately determine the available future paths ( e.g. , build orders in StarCraft ( Churchill & Buro , 2011 ) ) .

Summary: This paper studies the question of whether there is value in switching between exploration and exploitation behavior in RL. The authors investigate different signals for switching between learning a policy which maximizes an intrinsic reward (RND) and exploiting an extrinsic reward. The authors evaluate their approach on a number of Atari games, and show that an intra-episodic exploration scheme that switches based upon a "value discrepancy" is comparable/better to "episode" level exploration scheme on some evaluations. The authors additionally contribute some analysis showing the qualitative switching behavior and its effect on downstream performance.