When should agents explore?

Exploration remains a central challenge for reinforcement learning ( RL ) . Virtually all existing methods share the feature of a monolithic behaviour policy that changes only gradually ( at best ) . In contrast , the exploratory behaviours of animals and humans exhibit a rich diversity , namely including forms of switching between modes . This paper presents an initial study of mode-switching , nonmonolithic exploration for RL . We investigate different modes to switch between , at what timescales it makes sense to switch , and what signals make for good switching triggers . We also propose practical algorithmic components that make the switching mechanism adaptive and robust , which enables flexibility without an accompanying hyper-parameter-tuning burden . Finally , we report a promising and detailed analysis on Atari , using two-mode exploration and switching at sub-episodic time-scales . 1 INTRODUCTION . The trade-off between exploration and exploitation is described as the crux of learning and behaviour across many domains , not just reinforcement learning ( Sutton & Barto , 2018 ) , but also in decision making ( Cohen et al. , 2007 ) , evolutionary biology ( Cremer et al. , 2019 ) , ecology ( Kembro et al. , 2019 ) , neuroscience ( e.g. , focused versus diffuse search in visual attention ( Wolfe et al. , 1989 ) , dopamine regulations ( Chakroun et al. , 2020 ) ) , cognitive sciences ( Hills et al. , 2015 ) , as well as psychology and psychiatry ( Addicott et al. , 2017 ) . In a nutshell , exploration is about the balance between taking the familiar choice that is known to be rewarding and learning about unfamiliar options of uncertain reward , but which could ultimately be more valuable than the familiar options . Ample literature has studied the question of how much to explore , that is how to set the overall trade-off ( and how to adjust it over the course of learning ) ( Jaksch et al. , 2010 ; Cappé et al. , 2013 ; Lattimore & Szepesvári , 2020 ; Thrun , 1992 ) , and the question of how to explore , namely how to choose exploratory actions ( e.g. , randomly , optimistically , intrinsically motivated , or otherwise ) ( Schmidhuber , 1991 ; Oudeyer & Kaplan , 2009 ; Linke et al. , 2019 ) . In contrast , the question of when to explore has been studied very little , possibly because it does not arise in bandit problems , where a lot of exploration methods are rooted . The ‘ when ’ question and its multiple facets are the subjects of this paper . We believe that addressing it could lead to more intentional forms of exploration . Consider an agent that has access to two modes of behaviour , an ‘ explore ’ mode and an ‘ exploit ’ mode ( e.g. , a random policy and a greedy policy , as in ε-greedy ) . Even when assuming that the overall proportion of exploratory steps is fixed , the agent still has multiple degrees of freedom : it can explore more at the beginning of training and less in later phases ; it may take single exploratory steps or execute prolonged periods of exploration ; it may prefer exploratory steps early or late within an episode ; and it could trigger the onset ( or end ) of an exploratory period based on various criteria . Animals and humans exhibit non-trivial behaviour in all of these dimensions , presumably encoding useful inductive biases that way ( Power , 1999 ) . Humans make use of multiple effective strategies , such as selectively exploring options with high uncertainty ( a form of directed , or informationseeking exploration ) , and increasing the randomness of their choices when they are more uncertain ( Gershman , 2018 ; Gershman & Tzovaras , 2018 ; Ebitz et al. , 2019 ) . Monkeys use directed exploration to manage explore-exploit trade-offs , and these signals are coded in motivational brain regions ( Costa et al. , 2019 ) . Patients with schizophrenia register changes in directed exploration and experience low-grade inflammation when shifting from exploitation to random exploration ( Waltz et al. , 2020 ; Cathomas et al. , 2021 ) . This diversity is what motivates us to study which of these can benefit RL agents in turn , by expanding the class of exploratory behaviours beyond the commonly used monolithic ones ( where modes are merged homogeneously in time ) . 2 METHODS . The objective of an RL agent is to learn a policy that maximises external reward . At the high level , it achieves this by interleaving two processes : generating new experience by interacting with the environment using a behaviour policy ( exploration ) and updating its policy using this experience ( learning ) . As RL is applied to increasingly ambitious tasks , the challenge for exploration becomes to keep producing diverse experience , because if something has not been encountered , it can not be learned . Our central argument is therefore simple : a monolithic , time-homogeneous behaviour policy is strictly less diverse than a heterogeneous mode-switching one , and the former may hamstring the agent ’ s performance . As an illustrative example , consider a human learning how to ride a bike ( explore ) , while maintaining their usual happiness through food , sleep , work ( exploit ) : there is a stark contrast between a monolithic , time-homogeneous behaviour that interleaves a twist of the handlebar or a turn of a pedal once every few minutes or so , and the mode-switching behaviour that dedicates prolonged periods of time exclusively to acquiring the new skill of cycling . While the choice of behaviour in pure exploit mode is straightforward , namely the greedy pursuit of external reward ( or best guess thereof ) , denoted by G , there are numerous viable choices for behaviour in a pure explore mode ( denoted by X ) . In this paper we consider two standard ones : XU , the naive uniform random policy , and XI , an intrinsically motivated behaviour that exclusively pursues a novelty measure based on random network distillation ( RND , ( Burda et al. , 2018 ) ) . See Section 4 and Appendix B for additional possibilities of X . In this paper we choose fixed behaviours for these modes , and focus solely on the question of when to switch between them . In our setting , overall proportion of exploratory steps ( the how much ) , denoted by pX , is not directly controlled but derives from the when . 2.1 GRANULARITY . An exploration period is an uninterrupted sequence of steps in explore mode . We consider four choices of temporal granularity for exploratory periods , also illustrated in Figure 1 . Step-level exploration is the most common scenario , where the decision to explore is taken independently at each step , affecting one action.1 The canonical example is ε-greedy ( Fig.1 : C ) . Experiment-level exploration is the other extreme , where all behaviour during training is produced in explore mode , and learning is off-policy ( the greedy policy is only used for evaluation ) . This scenario is also very common , with most forms of intrinsic motivation falling into this category , namely pursuing reward with an intrinsic bonus throughout training ( Fig.1 : A ) .2 Episode-level exploration is the case where the mode is fixed for an entire episode at a time ( e.g. , training games versus tournament matches in a sport ) , see Fig.1 : B . This has been investigated for simple cases , where the policy ’ s level of stochasticity is sampled at the beginning of each episode ( Horgan et al. , 2018 ; Kapturowski et al. , 2019 ; Zha et al. , 2021 ) . Intra-episodic exploration is what falls in-between step- and episode-level exploration , where exploration periods last for multiple steps , but less than a full episode . This is the least commonly studied scenario , and will form the bulk of our investigations ( Fig.1 : D , E , F , G ) . We denote the length of an exploratory period by nX ( and similarly nG for exploit mode ) . To characterise granularity , our summary statistic of choice is medX : = median ( nX ) . Note that there are two possible units for these statistics : the raw steps or the proportion of the episode length L. The latter has different ( relative ) semantics , but may be more appropriate when episode lengths vary widely across training . We denote it as rmedX : = median ( nX /L ) . 1The length of an exploratory period tends to be short , but it can be greater than 1 , as multiple consecutive step-wise decisions to explore can create longer periods . 2Note that it is also possible to interpret ε-greedy as experiment-level exploration , where the X policy is fixed to a noisy version of G . 2.2 SWITCHING MECHANISMS . Granularity is but the coarsest facet of the ‘ when ’ question , but the more precise intra-episode timings matter too , namely when exactly to start and when to stop an exploratory period . This section introduces two mechanisms , blind and informed switching . It is worth highlighting that , in general , the mechanism ( or its time resolution ) for entering explore mode differs from the one for exiting it ( to enter exploit mode ) – this asymmetry is crucial to obtain flexible overall amounts of exploration . If switching were symmetric , the proportion would be pX ≈ 0.5 . Blind switching The simplest switching mechanism does not take any state into account ( thus , we call it blind ) and is only concerned with producing switches at some desired time resolution . It can be implemented deterministically through a counter ( e.g. , enter explore mode after 100 exploit mode steps ) , or probabilistically ( e.g. , at each step , enter explore mode with probability 0.01 ) . Its expected duration can be parameterised in terms of raw steps , or in terms of fractional episode length . The opposite of blind switching is informed switching , as discussed below . Informed switching Going beyond blind switching opens up another rich set of design choices , with switching informed by the agent ’ s internal state . There are two parts : first , a scalar trigger signal is produced by the agent at each step , using its current information – drawing inspiration from human behaviour , we view the triggering signal as a proxy for uncertainty ( Schulz et al. , 2019 ) : when uncertainty is high , the agent will switch to explore . Second , a binary switching decision is taken based on the trigger signal ( for example , by comparing it to a threshold ) . Again , triggering will generally not be symmetric between entering and exiting an exploratory period . To keep this paper focused , we will look at one such informed trigger , dubbed ‘ value promise discrepancy ’ ( see Appendix B for additional competitive variants ) . This is an online proxy of how much of the reward that the agent ’ s past value estimate promised ( k steps ago ) have actually come about . The intuition is that in uncertain parts of state space , this discrepancy will generally be larger than when everything goes as expected . Formally , Dpromise ( t− k , t ) : = ∣∣∣∣∣V ( st−k ) − k−1∑ i=0 γiRt−i − γkV ( st ) ∣∣∣∣∣ where V ( s ) is the agent ’ s value estimate at state s , R is the reward , and γ is a discount factor . Starting mode When periods last for a significant fraction of episode length , it also matters how the sequence is initialised , i.e. , whether an episode starts in explore or in exploit mode , or more generally , whether the agent explores more early in an episode or more later on . It is conceivable that the best choice among these is domain dependent ( see Figure 6 ) : in most scenarios , the states at the beginning of an episode have been visited many times , thus starting with exploit mode can be beneficial ; in other domains however , early actions may disproportionately determine the available future paths ( e.g. , build orders in StarCraft ( Churchill & Buro , 2011 ) ) .

Summary: This paper proposes to study exploration at different levels of granularity. Current methods either explore at the level of individual steps (e.g., \epsilon-greedy), or at the level of experiments (e.g. first a reward-free exploration phase, followed by a task-dependent learning phase using the gathered data). This paper proposes to study exploration at the intra-episodic level, i.e. where the agent switches between exploration and exploitation within the same episode.   They discuss various design choices to perform exploration at this level, for example switching after a certain number of steps or with a certain probability, or switching based on the discrepancy between the predicted value and actual experienced value.   The experimental results show that including intra-episodic exploration gives a modest benefit over other exploration schemes when using an R2D2 base agent. Other insights are also included, which show that the proportion of exploration does change throughout the learning process, indicating that different degrees of exploration are useful at different stages. They also show that the informed switching component learns switching behaviors which are non-uniform throughout the episodes.