Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems

1 INTRODUCTION . Deep reinforcement learning ( Deep RL ) has achieved landmark success in learning how to play Atari games from raw pixel input ( Mnih et al . ( 2015 ; 2016 ) ; Schaul et al . ( 2016 ) ; van Hasselt et al . ( 2016 ) ) . Many of these algorithms rely on naïve exploration techniques that employ random actions to find new rewards : either by selecting completely random actions at a fixed probability ( e.g . the ǫ-greedy methods of Q-learning ) , or by sampling stochastically from the current policy ’ s action distribution , as in policy gradient methods ( Mnih et al . ( 2016 ) ) . However , in sparse-reward games like Montezuma ’ s Revenge , no reward can be obtained at all until agents perform long sequences of complex behaviors such as descending ladders , crossing empty rooms , and avoiding enemies . Because the probability that any random action sequence will produce these behaviors is extremely low , a very large number of samples is required to find them ; and while it is now possible to obtain many Atari samples in reasonable wall-clock time ( Conti et al . ( 2018 ) ; Horgan et al . ( 2018 ) ; Salimans et al . ( 2017 ) ) , this approach is inefficient and will not scale well to harder tasks . For example , even after 22.8 billion Atari frames , Ape-X Horgan et al . ( 2018 ) ( a highly parallel , naïve-exploration algorithm ) can not match the performance achieved by 400 million frames of directed , count-based exploration ( Bellemare et al . ( 2016 ) ) on Montezuma ’ s Revenge ( Table 1 , Ape-X vs. PC ) . Sparsereward environments thus tend to require a prohibitively large number of training steps to solve without directed ( i.e . intelligent ) exploration ( Bellemare et al . ( 2016 ) ; Kulkarni et al . ( 2016 ) ; Tang et al . ( 2017 ) ) . Many approaches have tried to produce better exploration in RL ( Bellemare et al . ( 2016 ) ; Brafman & Tennenholtz ( 2001 ) ; Conti et al . ( 2018 ) ; Houthooft et al . ( 2018 ) ; Kearns & Singh ( 2002 ) ; Kulkarni et al . ( 2016 ) ; Oudeyer et al . ( 2007 ) ; Schmidhuber ( 2010 ) ; Tang et al . ( 2017 ) ) . These methods typically encourage agents to visit novel states or perform novel actions , requiring the algorithm to record how often states have been visited ( e.g . by maintaining a count , which is optimal in tabular RL ( Kolter & Ng ( 2009 ) ) , but requires further abstractions to scale to high-dimensional state spaces ( Bellemare et al . ( 2016 ) ; Tang et al . ( 2017 ) ) ) . Notably , these techniques record state visitations over the entirety of training time , requiring agents to express novelty in relation to all prior training episodes ( which we call across-training novelty ) . For example , in an algorithm based on pseudo-counts ( Bellemare et al . ( 2016 ) ) , the reward for visiting the same state in a future game is almost always lower than in the previous episode , encouraging agents to seek out states that have not been encountered in prior training iterations . Similarly , intrinsic motivation approaches that reward agents e.g . for visiting poorly-compressed states ( Schmidhuber ( 2010 ) ) tend to compress the same state better each time it is seen , meaning that future visits to that state may not be rewarded much ( or at all ) if the state has been visited in prior episodes . This observation also applies to the application of Novelty Search to Deep RL ( Conti et al . ( 2018 ) ; Lehman & Stanley ( 2011 ) ) , which calculates the novelty of a behavior by comparing it against an archive of prior agents ( Lehman & Stanley ( 2011 ) ) ; agents must perform new actions in relation to all their ancestors . Across-training approaches do not examine novelty within each lifetime ( called intra-life novelty ( Stanton & Clune ( 2016 ) ) ) . In our Atari experiments , we define a lifetime as the complete rollout for each game ( which can span multiple in-game lives ) , although we note that this definition is a hyperparameter choice . Making behaviors uninteresting solely because they have been performed in prior lifetimes can sometimes be detrimental : for example , it is beneficial for each human to learn their native language , even though many prior humans have already mastered that task . This may be especially important in multi-task environments ( Conti et al . ( 2018 ) ; Stanton & Clune ( 2016 ) ) . Consider a cross-shaped maze where to go in each cardinal direction , agents must master an entirely different skill ( swimming to go west , climbing mountains to go east , etc . ) and all skills must be acquired to solve a larger task ( Stanton & Clune ( 2016 ) ) . With across-training novelty , if an individual starts by exploring the west branch , it may continue to do so ( learning how to swim ) until state visitations become too high , and then explore a less-visited branch ( learning how to climb mountains ) where the exploration bonus is higher ( Conti et al . ( 2018 ) ; Stanton & Clune ( 2016 ) ) . Because visitations never decrease , the west branch ( and swimming behavior ) might never be revisited . The algorithm may explore the entire environment , but due to catastrophic forgetting ( French ( 1999 ) ) the final agent is likely a specialist ( Stanton & Clune ( 2016 ) ) ; it can perform only the most recent skill it learned and can not solve the larger task requiring all skills . Even if catastrophic forgetting can be eliminated , across-training novelty may not make progress if the required skills are hierarchical , or if a state visited too early ( offering no immediate reward ) is later critical for solving a task . For example , suppose an agent must open a door at the end of a hallway , but first requires a key located elsewhere in the environment . If the agent spends much too time initially exploring the hallway and only later discovers the key , it may not have any incentive to return to the hallway and the door-opening behavior might never be learned ( assuming state conflation i.e . that the hallway is still considered explored even after the key is obtained , which we expect would occur with deep neural network function approximators as it does with animals ) . Designing reward functions or state representations to avoid such pitfalls can be difficult ( Ng et al . ( 1999 ) ; Sutton & Barto ( 1998 ) ) , especially on any complex ( often deceptive ) task ( Nguyen et al . ( 2015 ) ; Woolley & Stanley ( 2011 ) ) . Experience replay does not alleviate this problem because it only helps once rewards have been discovered ; but we may need directed exploration to discover such rewards in the first place . While state counts could be decayed over time to refresh “ interest , ” choosing which states should be interesting again ( and when ) is a nontrivial , domain-specific task . An intra-life algorithm might avoid these issues because it conflates different exploration strategies . For example , an agent that first explores the hallway and then finds the key could receive identical intrinsic rewards as an agent that explores these states in reverse . Because both behaviors are similarly encouraged , it is more likely that learning will periodically switch between them and ultimately discover the greater value of obtaining the key first . Additionally , population-based search algorithms would not prefer one trajectory over the other ; an intra-life algorithm could thus preserve a diversity of different behaviors ( Stanton & Clune ( 2016 ) ) , making it easier to find the correct ordering since neither behavior is “ locked out ” by the history of exploration in prior episodes . We investigate the possibility of directly rewarding intra-life novelty : encouraging agents to explore as many states as possible within their lifetime . Using the previous example , even if the hallway is explored over many different games , an intra-life algorithm resets the exploration bonus such that the hallway becomes “ interesting ” again at the start of every new game . This is the approach taken by Curiosity Search ( Stanton & Clune ( 2016 ) ) , which promotes intra-life novelty by borrowing from biological principles of intrinsic motivation ( Berlyne ( 1960 ) ; Dayan & Belleine ( 2002 ) ; Kakade & Dayan ( 2000 ) ) . In vertebrates , this motivation is generally characterized by a dopamine response for behaviors that produce unexpected results ( i.e . a high temporal difference error ) ( Kakade & Dayan ( 2000 ) ) , encouraging the discovery of novel states and creating an instinct to play/explore that may facilitate an animal ’ s survival by leading to the acquisition of useful skills ( Benson-Amram & Holekamp ( 2012 ) ; Berlyne ( 1960 ) ; Kohler ( 1924 ) ; Ŝpinka et al . ( 2001 ) ) . Curiosity Search expresses this idea by first quantifying the types of rewarded behaviors and secondly rewarding agents for expressing as many of those behaviors as possible within each lifetime ( Stanton & Clune ( 2016 ) ) . An intra-life novelty compass can optionally be provided to direct agents towards novel behaviors and speed up learning , but is not required for the algorithm to function ( Stanton & Clune ( 2016 ) ) . Curiosity Search produced better overall performance than an across-training exploration algorithm ( Novelty Search ( Lehman & Stanley ( 2011 ) ) ) and improved the acquisition of door-opening skills in a maze environment ( Stanton & Clune ( 2016 ) ) . However , Curiosity Search was only investigated on a simple task and trained very small networks ( ∼100 parameters ) , in contrast to the millions of parameters that must be learned in Deep RL . In this work , we adapt Curiosity Search to high-dimensional , challenging RL games . Specifically , we discretize the pixel space of Atari games such as Montezuma ’ s Revenge into a curiosity grid and reward agents for visiting new locations on this grid . The grid is periodically reset such that every location becomes “ interesting ” again at the start of each game , thereby encouraging agents to express high intra-life novelty . We show that this adaptation , called DeepCS , matches the average performance of state-of-the-art methods on the hard-exploration game Montezuma ’ s Revenge and also performs well on Amidar , Freeway , Gravitar , Tutankham , and Seaquest ( in the latter of which DeepCS doubles A2C performance and achieves up to 80,000 points in one run ) . The strong performance of DeepCS on these sparse- and dense-reward tasks suggests that encouraging intra-life novelty is an interesting new approach for producing efficient domain exploration , and opens up new directions towards hybridizing across-training and intra-life exploration strategies . 2 METHODS . We primarily examine performance of DeepCS on two Atari games : Montezuma ’ s Revenge and Seaquest . The first is a notoriously difficult sparse-reward game in which exploration is critical to success . Naïve exploration strategies have trouble solving Montezuma ’ s Revenge due to the low frequency of rewards and the long sequences of complex behaviors required to obtain them ( Bellemare et al . ( 2016 ) ; Kulkarni et al . ( 2016 ) ; Tang et al . ( 2017 ) ) , making it an ideal candidate for demonstrating the benefits of intra-life exploration . Seaquest instead has a dense-reward space easily learned via traditional approaches ( Mnih et al . ( 2015 ; 2016 ) ; van Hasselt et al . ( 2016 ) ) , allowing us to examine the effects of intra-life exploration on games in which such techniques are not seemingly required . We also examine performance on 10 additional games : Alien , Amidar , Freeway , Gravitar , Kangaroo , Private Eye , Pitfall , Tutankham , Venture , and Wizard of Wor . Though not as infamously difficult as Montezuma ’ s Revenge , many of these games are hard-exploration tasks according to the taxonomy in ( Ostrovski et al . ( 2017 ) ) and are interesting challenges for intra-life exploration . We primarily focus our discussion on Montezuma ’ s Revenge because the latter has become a notoriously difficult challenge in RL ; only a handful of techniques ( Bellemare et al . ( 2016 ) ; Horgan et al . ( 2018 ) ; Kulkarni et al . ( 2016 ) ; Ostrovski et al . ( 2017 ) ; Tang et al . ( 2017 ) ) are able to reliably exit the first room ( achieving 400 points ) , in contrast to human players who explore many rooms and achieve a mean score of 4367 points ( Mnih et al . ( 2015 ) ) . In this work , we translate the general idea of Curiosity Search ( “ do something new ” ( Stanton & Clune ( 2016 ) ) ) into a reward for agents that “ go somewhere new ” in the game world . To do so , we divide the game world of Atari into a curiosity grid , each tile of which provides a reward when visited by the agent ( Fig . 1 ) . We then obtain the agent ’ s position from game RAM to indicate where on the grid an agent has visited ; if the game has multiple rooms , as in Montezuma ’ s Revenge , a distinct grid is created for each room . This approach is similar to the bitmasking of states performed in ( Kulkarni et al . ( 2016 ) ) with the main exception that we did not manually specify any objects of interest ( e.g . the key in Montezuma ’ s Revenge ) ; the grid is constructed from equally sized square tiles given an arbitrary grid size , discussed below . While we would ultimately prefer that agents learn to remember where they have been automatically from pixel input , extracting this information from RAM is simpler and allows us to directly examine whether encouraging intra-life novelty can help solve difficult exploration problems in RL . In future work , we could encourage the agent to learn directly from pixels where it has been ( or , more generally , what it has done ) e.g . with recurrent neural networks trained with an auxiliary task like outputting the curiosity grid per time-step . If one did not have access to RAM states , position could be inferred via object detection ( Fragkiadaki et al . ( 2015 ) ) or by implicit state-space representations such as Skip Context Tree Switching ( Bellemare et al . ( 2014 ) ) , the latter of which was shown to work well in a roughly similar counting-based model ( Bellemare et al . ( 2016 ) ) . Also interesting is investigating under what conditions such memories of past behaviors might emerge on their own because they are helpful for exploration ( e.g . in meta-learning ( Duan et al . ( 2018 ) ) ) . Agents were trained via A2C ( Mnih et al . ( 2016 ) ) using the default hyperparameters in OpenAI Baselines ( Dhariwal et al . ( 2017 ) ) . The clipped game reward at any state Rs was replaced with R̂s = βRs + ( 1 − β ) Is , where Is is the intrinsic reward ( 1 if an unexplored tile was just touched and 0 otherwise ) , Rs is the regular game reward clipped to [ −1 , 1 ] ( Mnih et al . ( 2015 ) ) , and β is a constant in the range [ 0 , 1 ] that controls the relative weight of intrinsic and regular game rewards . In the untuned case ( where both rewards are equal ) R̂s = clip ( Rs + Is ) . We chose β = 0.25 after a coarse hyperparameter sweep in Montezuma ’ s Revenge and used untuned values in all other games . Although we treat exploration as binary in this work , Is could also be reduced gradually in proportion to how many times a tile has been visited , similar to the mechanisms in other non-binary count-based approaches ( Bellemare et al . ( 2016 ) ; Brafman & Tennenholtz ( 2001 ) ; Kearns & Singh ( 2002 ) ; Ostrovski et al . ( 2017 ) ) . The intra-life novelty compass of Curiosity Search is traditionally a vector indicating the direction of the nearest rewards ( Stanton & Clune ( 2016 ) ) . Because direction is more appropriate for the first person view of a driving robot ( Lehman & Stanley ( 2011 ) ; Stanton & Clune ( 2016 ) ) and less so for Atari games ( in which the agent sees the entire “ world ” at once ) , we instead use the curiosity grid as an intra-life novelty compass , providing it as an extra network input . The agent must learn how to use this “ map ” to direct itself towards unexplored areas of the game . The curiosity grid is constructed using a per-tile grid size of 16x16 for most games , as described in SI Sec . S1 . To ensure dimensionality equal to normal game state inputs , the grid is resized to match the 84x84 downsampled game screen ( Mnih et al . ( 2015 ) ) . As discussed below , we found that this extra input was helpful , but not necessary to benefit DeepCS on hard RL tasks ; agents perform well if rewarded for visiting new areas even when they do not see where they have ( or have not ) already been ( Fig . 5 ) . All algorithms were trained for 160 million training steps with a frame skip of 4 ( Mnih et al . ( 2015 ) ) , resulting in 640 million game frames per run . 25 runs were performed for each experiment . Plotted results show medians ( solid lines ) , interquartile range ( shaded areas ) , and minimum/maximum scores ( dotted lines ) . We compare DeepCS ( with an underlying A2C implementation ) against A2C alone ; the latter is a synchronous and equally-performing implementation of A3C ( Mnih et al . ( 2016 ) ) . Our A2C algorithm uses 16 actors exploring in parallel ; training curves were obtained by examining the most recent episode score of each actor and plotting the averaged result . Statistical comparisons were obtained via a two-tailed Mann-Whitney U test , which does not assume normality . Videos of end-ofrun agents , open source code , and plot data can be obtained from : keptanonymousforreview . As with any exploration method , encouraging intra-life novelty has possible disadvantages . In a maze environment , Curiosity Search will reward agents for exploring every dead end , even though such states are likely useless for solving the maze itself . Our hypothesis is not that the intra-life exploration of Curiosity Search is superior to that of across-training novelty , but rather that the former is a valuable and interesting tool for solving challenging exploration problems in RL . We also note that while DeepCS might encourage inefficient exploration , ( 1 ) this may not prove problematic as the algorithm still values real game rewards ( how much is a hyperparameter choice ) and may find higher ultimate value in ignoring some dead ends , and ( 2 ) one could use the exploration produced by DeepCS to later produce a distilled , exploitive , efficient policy ( e.g . by annealing the weight on intrinsic rewards at the end of training or filling an experience replay buffer with states observed via DeepCS agents and then training a DQN agent off-policy to only maximize real game rewards ) .

Summary: This paper proposes use of intra-life coverage (an agent must visit all locations within each episode) for effective exploration in Atari games. This is in contrast of approaches that use inter-life coverage or curiosity metrics to incentivize exploration. The paper shows detailed results and analysis on 2 Atari games: Montezuma’s Revenge and Seaquest, and reports results on other games as well.