Explicit Recall for Efficient Exploration

1 INTRODUCTION . Incorporating deep neural networks ( DNNs ) has brought significant advance to reinforcement learning ( Mnih et al. , 2015 ; 2016 ) . In addition to the ability to approximate nonlinear functions , convolutional neural networks ( CNNs ) provide an effective way to model local patterns , thus enabling the agent to learn from high-dimension visual inputs ( e.g. , Atari 2600 games ( Mnih et al. , 2015 ) ) or even combinatorial discrete spaces ( e.g. , the game of Go ( Silver et al. , 2017 ) ) . However , as the scale of problems increases , efficient exploration becomes even more challenging . For instance , in the game Montezuma ’ s Revenge ( Bellemare et al. , 2013 ) , the player moves in a large partially-observable maze , searches for props and fights with monsters . Nonzero rewards come from successfully obtaining props or killing monsters that involve a long sequence of actions . As a result , nonzero reward signals in these problems are very sparse , and simple exploration algorithms like -greedy are likely to fail . How to exploit the power of DNNs for efficient exploration has become an active research field in Deep Reinforcement Learning ( DRL ) . Sharing the same spirit of encouraging novelty , the majority of works in this track can be roughly divided into two groups . The first group of works , partially motivated by cognitive sciences , introduces the concept of novelty about states ( Houthooft et al. , 2016 ; Bellemare et al. , 2016 ; Pathak et al. , 2017 ) . The agent will get extra “ intrinsic ” rewards when it explores “ novel ” states following some metrics . This is also called “ curiosity-driven ” exploration in some literatures ( Pathak et al. , 2017 ) . The second group of work aims at introducing “ goal-driven ” exploration . The agent cooperates with a goal generator ( either human-programmed oracles ( Baranes & Oudeyer , 2013 ) or generators jointly optimized with the agent ( Held et al. , 2017 ; Sukhbaatar et al. , 2017 ) ) . Given such internal goals , the agent will get rewards when it manages to reach the goal . As the difficulty in reaching goals increases , the agent will learn more skills and explore more states that it could not reach before , leading to self-paced learning or curriculum learning ( Held et al. , 2017 ) . In this paper , we study another important factor in exploration — memory , which has received little attention in the past . In particular , we introduce Explicit Recall , a general mechanism for efficient exploration that can be naturally combined with many existing exploration algorithms , including several that have been mentioned above . The observed trajectories are explicitly memorized in a prioritized data structure . Different from typical DRL algorithms which memorize the value ( novelty ) of states implicitly in a black-box DNN and compose policies to explore the novel states , in Explicit Recall , the exploration trials are based on the recall of observed trajectories leading to previously visited states with high novelty . Inspired by cognitive science findings that memorization of observed trajectories helps human to reach previously visited states ( Thirkettle et al. , 2013 ) , we study whether and how explicit recall improves the exploration in DRL . We study the performance of Explicit Recall on two typical exploration tasks : knowledge/skill learning and online exploration/exploitation tradeoff . The first focuses more on the pure exploration and aims at learning knowledge ( e.g. , the model of the environment ) or skills ( e.g. , the capability of reaching goal states ) during the exploration process . The second focuses on balancing exploration and exploiting in a more typical reinforcement learning setting , with the goal of maximizing longterm rewards . 2 EXPLICIT RECALL . Explicit Recall integrates explicit memory of observed trajectories into deep reinforcement learning . The exploration of the agent begins with the recall of a trajectory leading to a previously visited state that is considered to be novel . This is based on the assumption that : states next to novel states are usually novel states , where the state locality is defined by the state transition function . Although the validation of this assumption depends on both the metric used for measuring novelty and the environment , we empirically demonstrate the performance of models that rely on this assumption in challenging simulated environments . With the recalled trajectory , the agent first follows the trajectory to reach the end state of the trajectory ( a novel state ) . The real exploration of the environment then begins . The trajectory of this new exploration trial gets stored in the memory . To make the exposition concrete , in most parts of the paper , we adapt the state novelty definition from curiosity-driven RL ( Pathak et al. , 2017 ) . It should be noted that Explicit Recall may be used in combination with other exploration algorithms as well , as demonstrated later . As in previous work of Pathak et al . ( 2017 ) , we have a dynamics function ψ that is trained for the environment based on agents ’ experience . The agents also learns an inverse dynamics function φ — predicting the state transition in the feature space of ψ . The prediction error is treated as the novelty for states . One can easily choose other metrics such as psuedo-count ( Bellemare et al. , 2016 ) . From the view of exploration , this approach makes the agent focus more on pure exploration but less on how to reach novel states . For better intuition , imagine that you live in a village located at the center of a big map . The whole map can be divided into multiple regions . The central area of your village is what you are very familiar with . On the edge of the village are the places where you may have previously visited but not so often , and thus are unfamiliar . The places beyond the edge of the village are what you do not know about at all . Your exploration should begin with reaching the border of the village , and then explore the outer world . Analogous to this idea , the agent begins its exploration from a novel state ( on the edge ) by following a memorized trajectory . The idea bears some similarity to the E3 algorithm of Kearns & Singh ( 1998 ) , although they focus on finite-state MDPs and can use an exact planner ( such as value iteration ) to reach novel states . Extending E3 or the related Rmax algorithm ( Brafman & Tennenholtz , 2001 ) to deep RL is tricky : instead of using tabular planning algorithms , one must use complex blackbox models , such as a DQN ( Mnih et al. , 2015 ) or a policy network ( Mnih et al. , 2016 ) , to encode learned policies to reach novel states . In contrast , our method memorizes trajectories to reach novel states , which is a simple and transparent process . 2.1 EXPLICIT TRAJECTORY MEMORY . The trajectory is organized as a tuple consisting a sequence of states { si } ki=1 and the actions { ai } k=1i=1 along this k-length trajectory traj , and a real-valued score score ( traj ) for this trajectory . Since we are interested in the novelty of the state that this trajectory is leading to , we choose the novelty of the end state novelty ( sk ) as the score . If we are also considering environmental ( extrinsic reward ) , a mixture of extrinsic rewards and the novelty is used : score ( traj ) = novelty ( sk ) + λ stage avg reward ( traj ) ( 1 ) where λ is a task-dependent hyper-parameter and stage avg reward is a function similar to average extrinsic reward function over the trajectory , see details in Appendix B . We construct the algorithm based on the following assumption : states next to novel states are usually novel states . Thus , we give priority to trajectories with high scores . We organize the trajectory memory as a prioritized pool pool with a fixed capacity capacity ( pool ) . As mentioned before , during the most recent exploration trial , new trajectories are recorded periodically and added to the memory . When the size of the pool exceeds its capacity , the trajectory with smallest score ( traj ) will be removed from the pool . 2.2 EXPLORATION WITH TRAJECTORY RECALL . Exploration starts from choosing places of interest based on the memory . The trajectory with highest score is selected ( and then removed ) from the pool . The end point of the trajectory is the target ( goal ) state to be reached . We discuss the problem of reaching the target state based on the recalled trajectory in the next section . After reaching the state , the agent will begin its exploration . In this phase , one can incorporate any existing exploration algorithms . A naive way for exploration is to perform random actions , which actually , shows acceptable results in our empirical study . We discuss several choices of the exploration method in Section 3.1 . 2.3 STATE REACHING . We now consider the problem of reaching a target state introduced by a trajectory . For simplicity , we assume that the environment always starts from the same initial state , which is a typical setting in reinforcement learning . If the state transition is completely deterministic , we can directly perform the recalled action sequence of the trajectory { ai } and reach the designated target . However , this condition is not satisfied in many real-world applications due to the noise or partial observability of the environment . To resolve the state reaching problem with recalled trajectory , we incorporate a PATH function . Given a ( starting state , target state ) pair , PATH function outputs a probability distribution over action space for the first action to take at state s in order to reach state s′ . Thus , we can generate a path from current state to the goal state by iteratively invoking the PATH function . The pseudo-code of the full algorithm for state reaching is demonstrated in Algorithm 1 in Appendix C. In short , we iteratively pick states from the trajectory as a sub-goal . PATH function is invoked to reach this sub-goal . The step-distance in the trajectory between the current state and the sub-goal is much less than the length of the whole trajectory . When the agent reaches the sub-goal , we will randomly pick a next sub-goal for the agent . The advantage of this procedure is that it explicitly decomposes a long-term goal into short-term sub-goals . To determine whether the sub-goal state has been reached or not , we measure the cosine distance between the current state and the sub-goal in feature space introduced by the inverse dynamics φ . When the similarity exceeds a hyper-parameter thresh , we treat two states to be the same . The PATH function is jointly optimized with the agent . We treat the learning of PATH function as another reinforcement learning problem . If the agent successfully reaches a designated sub-goal ( i.e. , a path from the starting state to the target state has been found ) , the agent will be rewarded . The reward is determined by the cosine distance between the actual ending state and the sub-goal . Moreover , we give small penalty to the agent at each step to encourage it to reach the sub-goal as fast as possible .

Summary: In this paper, the authors propose an exploration strategy based on the explicit storage and recall of trajectories leading to novel states. A pool of such trajectories is managed over time, and a method is proposed so that the agent can learn how to follow a path corresponding to these trajectories so as to explore novel states. The idea is demonstrated in a set of room experiments, and quickly shown efficient in Montezuma's Revenge and PrivateEye Atari games.