Never Give Up: Learning Directed Exploration Strategies

1 INTRODUCTION . The problem of exploration remains one of the major challenges in deep reinforcement learning . In general , methods that guarantee finding an optimal policy require the number of visits to each state–action pair to approach infinity . Strategies that become greedy after a finite number of steps may never learn to act optimally ; they may converge prematurely to suboptimal policies , and never gather the data they need to learn . Ensuring that all state-action pairs are encountered infinitely often is the general problem of maintaining exploration ( François-Lavet et al. , 2018 ; Sutton & Barto , 2018 ) . The simplest approach for tackling this problem is to consider stochastic policies with a non-zero probability of selecting all actions in each state , e.g . -greedy or Boltzmann exploration . While these techniques will eventually learn the optimal policy in the tabular setting , they are very inefficient and the steps they require grow exponentially with the size of the state space . Despite these shortcomings , they can perform remarkably well in dense reward scenarios ( Mnih et al. , 2015 ) . In sparse reward settings , however , they can completely fail to learn , as temporally-extended exploration ( also called deep exploration ) is crucial to even find the very few rewarding states ( Osband et al. , 2016 ) . ∗Equal contribution . Recent approaches have proposed to provide intrinsic rewards to agents to drive exploration , with a focus on demonstrating performance in non-tabular settings . These intrinsic rewards are proportional to some notion of saliency quantifying how different the current state is from those already visited ( Bellemare et al. , 2016 ; Haber et al. , 2018 ; Houthooft et al. , 2016 ; Oh et al. , 2015 ; Ostrovski et al. , 2017 ; Pathak et al. , 2017 ; Stadie et al. , 2015 ) . As the agent explores the environment and becomes familiar with it , the exploration bonus disappears and learning is only driven by extrinsic rewards . This is a sensible idea as the goal is to maximise the expected sum of extrinsic rewards . While very good results have been achieved on some very hard exploration tasks , these algorithms face a fundamental limitation : after the novelty of a state has vanished , the agent is not encouraged to visit it again , regardless of the downstream learning opportunities it might allow ( Bellemare et al. , 2016 ; Ecoffet et al. , 2019 ; Stanton & Clune , 2018 ) . Other methods estimate predictive forward models ( Haber et al. , 2018 ; Houthooft et al. , 2016 ; Oh et al. , 2015 ; Pathak et al. , 2017 ; Stadie et al. , 2015 ) and use the prediction error as the intrinsic motivation . Explicitly building models like this , particularly from observations , is expensive , error prone , and can be difficult to generalize to arbitrary environments . In the absence of the novelty signal , these algorithms reduce to undirected exploration schemes , maintaining exploration in a non-scalable way . To overcome this problem , a careful calibration between the speed of the learning algorithm and that of the vanishing rewards is required ( Ecoffet et al. , 2019 ; Ostrovski et al. , 2017 ) . The main idea of our proposed approach is to jointly learn separate exploration and exploitation policies derived from the same network , in such a way that the exploitative policy can concentrate on maximising the extrinsic reward ( solving the task at hand ) while the exploratory ones can maintain exploration without eventually reducing to an undirected policy . We propose to jointly learn a family of policies , parametrised using the UVFA framework ( Schaul et al. , 2015a ) , with various degrees of exploratory behaviour . The learning of the exploratory policies can be thought of as a set of auxiliary tasks that can help build a shared architecture that continues to develop even in the absence of extrinsic rewards ( Jaderberg et al. , 2016 ) . We use reinforcement learning to approximate the optimal value function corresponding to several different weightings of intrinsic rewards . We propose an intrinsic reward that combines per-episode and life-long novelty to explicitly encourage the agent to repeatedly visit all controllable states in the environment over an episode . Episodic novelty encourages an agent to periodically revisit familiar ( but potentially not fully explored ) states over several episodes , but not within the same episode . Life-long novelty gradually down-modulates states that become progressively more familiar across many episodes . Our episodic novelty uses an episodic memory filled with all previously visited states , encoded using the self-supervised objective of Pathak et al . ( 2017 ) to avoid uncontrollable parts of the state space . Episodic novelty is then defined as similarity of the current state to previously stored states . This allows the episodic novelty to rapidly adapt within an episode : every observation made by the agent potentially changes the per-episode novelty significantly . Our life-long novelty multiplicatively modulates the episodic similarity signal and is driven by a Random Network Distillation error ( Burda et al. , 2018b ) . In contrast to the episodic novelty , the life-long novelty changes slowly , relying upon gradient descent optimisation ( as opposed to an episodic memory write for episodic novelty ) . Thus , this combined notion of novelty is able to generalize in complex tasks with large , high dimensional state spaces in which a given state is never observed twice , and maintain consistent exploration both within an episode and across episodes . This paper makes the following contributions : ( i ) defining an exploration bonus combining life-long and episodic novelty to learn exploratory strategies that can maintain exploration throughout the agent ’ s training process ( to never give up ) , ( ii ) to learn a family of policies that separate exploration and exploitation using a conditional architecture with shared weights , ( iii ) experimental evidence that the proposed method is scalable and performs on par or better than state-of-the-art methods on hard exploration tasks . Our work differs from Savinov et al . ( 2018 ) in that it is not specialised to navigation tasks , our method incorporates a long-term intrinsic reward and is able to separate exploration and exploitation policies . Unlike Stanton & Clune ( 2018 ) , our work relies on no privileged information and combines both episodic and non-episodic novelty , obtaining superior results . Our work differs from Beyer et al . ( 2019 ) in that we learn multiple policies by sharing weights , rather than just a common replay buffer , and our method does not require exact counts and so can scale to more realistic domains such as Atari . The paper is organized as follows . In Section 2 we describe the proposed intrinsic reward . In Section 3 , we describe the proposed agent and general framework . In Section 4 we present experimental evaluation . 2 THE NEVER-GIVE-UP INTRINSIC REWARD . We follow the literature on curiosity-driven exploration , where the extrinsic reward is augmented with an intrinsic reward ( or exploration bonus ) . The augmented reward at time t is then defined as rt = r e t + βr i t , where r e t and r i t are respectively the extrinsic and intrinsic rewards , and β is a positive scalar weighting the relevance of the latter . Deep RL agents are typically trained on the augmented reward rt , while performance is measured on extrinsic reward ret only . This section describes the proposed intrinsic reward rit . Our intrinsic reward rit satisfies three properties : ( i ) it rapidly discourages revisiting the same state within the same episode , ( ii ) it slowly discourages visits to states visited many times across episodes , ( iii ) the notion of state ignores aspects of an environment that are not influenced by an agent ’ s actions . We begin by providing a general overview of the computation of the proposed intrinsic reward . Then we provide the details of each one of the components . The reward is composed of two blocks : an episodic novelty module and an ( optional ) life-long novelty module , represented in red and green respectively in Fig . 1 ( right ) . The episodic novelty module computes our episodic intrinsic reward and is composed of an episodic memory , M , and an embedding function f , mapping the current observation to a learned representation that we refer to as controllable state . At the beginning of each episode , the episodic memory starts completely empty . At every step , the agent computes an episodic intrinsic reward , repisodict , and appends the controllable state corresponding to the current observation to the memory M . To determine the bonus , the current observation is compared to the content of the episodic memory . Larger differences produce larger episodic intrinsic rewards . The episodic intrinsic reward repisodict promotes the agent to visit as many different states as possible within a single episode . This means that the notion of novelty ignores inter-episode interactions : a state that has been visited thousands of times gives the same intrinsic reward as a completely new state as long as they are equally novel given the history of the current episode . A life-long ( or inter-episodic ) novelty module provides a long-term novelty signal to statefully control the amount of exploration across episodes . We do so by multiplicatively modulating the exploration bonus repisodict with a life-long curiosity factor , αt . Note that this modulation will vanish over time , reducing our method to using the non-modulated reward . Specifically , we combine αt with r episodic t as follows ( see also Fig . 1 ( right ) ) : rit = r episodic t ·min { max { αt , 1 } , L } ( 1 ) where L is a chosen maximum reward scaling ( we fix L = 5 for all our experiments ) . Mixing rewards this way , we leverage the long-term novelty detection that αt offers , while rit continues to encourage our agent to explore all the controllable states . Embedding network : f : O → Rp maps the current observation to a p-dimensional vector corresponding to its controllable state . Consider an environment that has a lot of variability independent of the agent ’ s actions , such as navigating a busy city with many pedestrians and vehicles . An agent could visit a large number of different states ( collecting large cumulative intrinsic rewards ) without taking any actions . This would not lead to performing any meaningful form of exploration . To avoid such meaningless exploration , given two consecutive observations , we train a Siamese network ( Bromley et al. , 1994 ; Koch et al. , 2015 ) f to predict the action taken by the agent to go from one observation to the next ( Pathak et al. , 2017 ) . Intuitively , all the variability in the environment that is not affected by the action taken by the agent would not be useful to make this prediction . More formally , given a triplet { xt , at , xt+1 } composed of two consecutive observations , xt and xt+1 , and the action taken by the agent at , we parameterise the conditional likelihood as p ( a|xt , xt+1 ) = h ( f ( xt ) , f ( xt+1 ) ) , where h is a one hidden layer MLP followed by a softmax . The parameters of both h and f are trained via maximum likelihood . This architecture can be thought of as a Siamese network with a one-layer classifier on top , see Fig . 1 ( left ) for an illustration . For more details about the architecture , see App . H.1 , and hyperparameters , see App . F. Episodic memory and intrinsic reward : The episodic memory M is a dynamically-sized slotbased memory that stores the controllable states in an online fashion ( Pritzel et al. , 2017 ) . At time t , the memory contains the controllable states of all the observations visited in the current episode , { f ( x0 ) , f ( x1 ) , . . . , f ( xt−1 ) } . Inspired by theoretically-justified exploration methods turning stateaction counts into a bonus reward ( Strehl & Littman , 2008 ) , we define our intrinsic reward as repisodict = 1√ n ( f ( xt ) ) ≈ 1√∑ fi∈Nk K ( f ( xt ) , fi ) + c ( 2 ) where n ( f ( xt ) ) is the counts for the visits to the abstract state f ( xt ) . We approximate these counts n ( f ( xt ) ) as the sum of the similarities given by a kernel functionK : Rp×Rp → R , over the content of M . In practice , pseudo-counts are computed using the k-nearest neighbors of f ( xt ) in the memory M , denoted by Nk = { fi } ki=1 . The constant c guarantees a minimum amount of “ pseudo-counts ” ( fixed to 0.001 in all our experiments ) . Note that when K is a Dirac delta function , the approximation becomes exact but consequently provides no generalisation of exploration required for very large state spaces . Following Blundell et al . ( 2016 ) ; Pritzel et al . ( 2017 ) , we use the inverse kernel for K , K ( x , y ) = d2 ( x , y ) d2m + ( 3 ) where is a small constant ( fixed to 10−3 in all our experiments ) , d is the Euclidean distance and d2m is a running average of the squared Euclidean distance of the k-th nearest neighbors . This running average is used to make the kernel more robust to the task being solved , as different tasks may have different typical distances between learnt embeddings . A detailed computation of the episodic reward can be found in Alg . 1 in App . A.1 . Integrating life-long curiosity : In principle , any long-term novelty estimator could be used as a basis for the modulator αt . We found Random Network Distillation ( Burda et al. , 2018b , RND ) worked well , is simple to implement and easy to parallelize . The RND modulator αt is defined by introducing a random , untrained convolutional network g : O → Rk , and training a predictor network ĝ : O → Rk that attempts to predict the outputs of g on all the observations that are seen during training by minimizing err ( xt ) = ||ĝ ( xt ; θ ) − g ( xt ) ||2 with respect to the parameters of ĝ , θ . We then define the modulator αt as a normalized mean squared error , as done in Burda et al . ( 2018b ) : αt = 1 + err ( xt ) −µe σe , where σe and µe are running standard deviation and mean for err ( xt ) . For more details about the architecture , see App . H.2 , and hyperparameters , see App . F .

Summary: The paper proposes a novel intrinsic reward/curiosity metric that combines both episodic and “life-long” novelty. Essentially two competing pressures that push agents to explore as many novel states in a single rollout as possible and to explore as many states as possible as evenly as possible. The primary contribution here is the episodic novelty measure, which relies on a state embedding that takes into account stochasticity in the environment. The paper covers this episodic curiosity measure and how it’s integrated with the life-long curiosity metric. It then demonstrates the impact of these metrics and variations compared to baselines on particular games and all 57 Arcade Learning Environment games.