Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems

1 INTRODUCTION . Deep reinforcement learning ( Deep RL ) has achieved landmark success in learning how to play Atari games from raw pixel input ( Mnih et al . ( 2015 ; 2016 ) ; Schaul et al . ( 2016 ) ; van Hasselt et al . ( 2016 ) ) . Many of these algorithms rely on naïve exploration techniques that employ random actions to find new rewards : either by selecting completely random actions at a fixed probability ( e.g . the ǫ-greedy methods of Q-learning ) , or by sampling stochastically from the current policy ’ s action distribution , as in policy gradient methods ( Mnih et al . ( 2016 ) ) . However , in sparse-reward games like Montezuma ’ s Revenge , no reward can be obtained at all until agents perform long sequences of complex behaviors such as descending ladders , crossing empty rooms , and avoiding enemies . Because the probability that any random action sequence will produce these behaviors is extremely low , a very large number of samples is required to find them ; and while it is now possible to obtain many Atari samples in reasonable wall-clock time ( Conti et al . ( 2018 ) ; Horgan et al . ( 2018 ) ; Salimans et al . ( 2017 ) ) , this approach is inefficient and will not scale well to harder tasks . For example , even after 22.8 billion Atari frames , Ape-X Horgan et al . ( 2018 ) ( a highly parallel , naïve-exploration algorithm ) can not match the performance achieved by 400 million frames of directed , count-based exploration ( Bellemare et al . ( 2016 ) ) on Montezuma ’ s Revenge ( Table 1 , Ape-X vs. PC ) . Sparsereward environments thus tend to require a prohibitively large number of training steps to solve without directed ( i.e . intelligent ) exploration ( Bellemare et al . ( 2016 ) ; Kulkarni et al . ( 2016 ) ; Tang et al . ( 2017 ) ) . Many approaches have tried to produce better exploration in RL ( Bellemare et al . ( 2016 ) ; Brafman & Tennenholtz ( 2001 ) ; Conti et al . ( 2018 ) ; Houthooft et al . ( 2018 ) ; Kearns & Singh ( 2002 ) ; Kulkarni et al . ( 2016 ) ; Oudeyer et al . ( 2007 ) ; Schmidhuber ( 2010 ) ; Tang et al . ( 2017 ) ) . These methods typically encourage agents to visit novel states or perform novel actions , requiring the algorithm to record how often states have been visited ( e.g . by maintaining a count , which is optimal in tabular RL ( Kolter & Ng ( 2009 ) ) , but requires further abstractions to scale to high-dimensional state spaces ( Bellemare et al . ( 2016 ) ; Tang et al . ( 2017 ) ) ) . Notably , these techniques record state visitations over the entirety of training time , requiring agents to express novelty in relation to all prior training episodes ( which we call across-training novelty ) . For example , in an algorithm based on pseudo-counts ( Bellemare et al . ( 2016 ) ) , the reward for visiting the same state in a future game is almost always lower than in the previous episode , encouraging agents to seek out states that have not been encountered in prior training iterations . Similarly , intrinsic motivation approaches that reward agents e.g . for visiting poorly-compressed states ( Schmidhuber ( 2010 ) ) tend to compress the same state better each time it is seen , meaning that future visits to that state may not be rewarded much ( or at all ) if the state has been visited in prior episodes . This observation also applies to the application of Novelty Search to Deep RL ( Conti et al . ( 2018 ) ; Lehman & Stanley ( 2011 ) ) , which calculates the novelty of a behavior by comparing it against an archive of prior agents ( Lehman & Stanley ( 2011 ) ) ; agents must perform new actions in relation to all their ancestors . Across-training approaches do not examine novelty within each lifetime ( called intra-life novelty ( Stanton & Clune ( 2016 ) ) ) . In our Atari experiments , we define a lifetime as the complete rollout for each game ( which can span multiple in-game lives ) , although we note that this definition is a hyperparameter choice . Making behaviors uninteresting solely because they have been performed in prior lifetimes can sometimes be detrimental : for example , it is beneficial for each human to learn their native language , even though many prior humans have already mastered that task . This may be especially important in multi-task environments ( Conti et al . ( 2018 ) ; Stanton & Clune ( 2016 ) ) . Consider a cross-shaped maze where to go in each cardinal direction , agents must master an entirely different skill ( swimming to go west , climbing mountains to go east , etc . ) and all skills must be acquired to solve a larger task ( Stanton & Clune ( 2016 ) ) . With across-training novelty , if an individual starts by exploring the west branch , it may continue to do so ( learning how to swim ) until state visitations become too high , and then explore a less-visited branch ( learning how to climb mountains ) where the exploration bonus is higher ( Conti et al . ( 2018 ) ; Stanton & Clune ( 2016 ) ) . Because visitations never decrease , the west branch ( and swimming behavior ) might never be revisited . The algorithm may explore the entire environment , but due to catastrophic forgetting ( French ( 1999 ) ) the final agent is likely a specialist ( Stanton & Clune ( 2016 ) ) ; it can perform only the most recent skill it learned and can not solve the larger task requiring all skills . Even if catastrophic forgetting can be eliminated , across-training novelty may not make progress if the required skills are hierarchical , or if a state visited too early ( offering no immediate reward ) is later critical for solving a task . For example , suppose an agent must open a door at the end of a hallway , but first requires a key located elsewhere in the environment . If the agent spends much too time initially exploring the hallway and only later discovers the key , it may not have any incentive to return to the hallway and the door-opening behavior might never be learned ( assuming state conflation i.e . that the hallway is still considered explored even after the key is obtained , which we expect would occur with deep neural network function approximators as it does with animals ) . Designing reward functions or state representations to avoid such pitfalls can be difficult ( Ng et al . ( 1999 ) ; Sutton & Barto ( 1998 ) ) , especially on any complex ( often deceptive ) task ( Nguyen et al . ( 2015 ) ; Woolley & Stanley ( 2011 ) ) . Experience replay does not alleviate this problem because it only helps once rewards have been discovered ; but we may need directed exploration to discover such rewards in the first place . While state counts could be decayed over time to refresh “ interest , ” choosing which states should be interesting again ( and when ) is a nontrivial , domain-specific task . An intra-life algorithm might avoid these issues because it conflates different exploration strategies . For example , an agent that first explores the hallway and then finds the key could receive identical intrinsic rewards as an agent that explores these states in reverse . Because both behaviors are similarly encouraged , it is more likely that learning will periodically switch between them and ultimately discover the greater value of obtaining the key first . Additionally , population-based search algorithms would not prefer one trajectory over the other ; an intra-life algorithm could thus preserve a diversity of different behaviors ( Stanton & Clune ( 2016 ) ) , making it easier to find the correct ordering since neither behavior is “ locked out ” by the history of exploration in prior episodes . We investigate the possibility of directly rewarding intra-life novelty : encouraging agents to explore as many states as possible within their lifetime . Using the previous example , even if the hallway is explored over many different games , an intra-life algorithm resets the exploration bonus such that the hallway becomes “ interesting ” again at the start of every new game . This is the approach taken by Curiosity Search ( Stanton & Clune ( 2016 ) ) , which promotes intra-life novelty by borrowing from biological principles of intrinsic motivation ( Berlyne ( 1960 ) ; Dayan & Belleine ( 2002 ) ; Kakade & Dayan ( 2000 ) ) . In vertebrates , this motivation is generally characterized by a dopamine response for behaviors that produce unexpected results ( i.e . a high temporal difference error ) ( Kakade & Dayan ( 2000 ) ) , encouraging the discovery of novel states and creating an instinct to play/explore that may facilitate an animal ’ s survival by leading to the acquisition of useful skills ( Benson-Amram & Holekamp ( 2012 ) ; Berlyne ( 1960 ) ; Kohler ( 1924 ) ; Ŝpinka et al . ( 2001 ) ) . Curiosity Search expresses this idea by first quantifying the types of rewarded behaviors and secondly rewarding agents for expressing as many of those behaviors as possible within each lifetime ( Stanton & Clune ( 2016 ) ) . An intra-life novelty compass can optionally be provided to direct agents towards novel behaviors and speed up learning , but is not required for the algorithm to function ( Stanton & Clune ( 2016 ) ) . Curiosity Search produced better overall performance than an across-training exploration algorithm ( Novelty Search ( Lehman & Stanley ( 2011 ) ) ) and improved the acquisition of door-opening skills in a maze environment ( Stanton & Clune ( 2016 ) ) . However , Curiosity Search was only investigated on a simple task and trained very small networks ( ∼100 parameters ) , in contrast to the millions of parameters that must be learned in Deep RL . In this work , we adapt Curiosity Search to high-dimensional , challenging RL games . Specifically , we discretize the pixel space of Atari games such as Montezuma ’ s Revenge into a curiosity grid and reward agents for visiting new locations on this grid . The grid is periodically reset such that every location becomes “ interesting ” again at the start of each game , thereby encouraging agents to express high intra-life novelty . We show that this adaptation , called DeepCS , matches the average performance of state-of-the-art methods on the hard-exploration game Montezuma ’ s Revenge and also performs well on Amidar , Freeway , Gravitar , Tutankham , and Seaquest ( in the latter of which DeepCS doubles A2C performance and achieves up to 80,000 points in one run ) . The strong performance of DeepCS on these sparse- and dense-reward tasks suggests that encouraging intra-life novelty is an interesting new approach for producing efficient domain exploration , and opens up new directions towards hybridizing across-training and intra-life exploration strategies . 2 METHODS . We primarily examine performance of DeepCS on two Atari games : Montezuma ’ s Revenge and Seaquest . The first is a notoriously difficult sparse-reward game in which exploration is critical to success . Naïve exploration strategies have trouble solving Montezuma ’ s Revenge due to the low frequency of rewards and the long sequences of complex behaviors required to obtain them ( Bellemare et al . ( 2016 ) ; Kulkarni et al . ( 2016 ) ; Tang et al . ( 2017 ) ) , making it an ideal candidate for demonstrating the benefits of intra-life exploration . Seaquest instead has a dense-reward space easily learned via traditional approaches ( Mnih et al . ( 2015 ; 2016 ) ; van Hasselt et al . ( 2016 ) ) , allowing us to examine the effects of intra-life exploration on games in which such techniques are not seemingly required . We also examine performance on 10 additional games : Alien , Amidar , Freeway , Gravitar , Kangaroo , Private Eye , Pitfall , Tutankham , Venture , and Wizard of Wor . Though not as infamously difficult as Montezuma ’ s Revenge , many of these games are hard-exploration tasks according to the taxonomy in ( Ostrovski et al . ( 2017 ) ) and are interesting challenges for intra-life exploration . We primarily focus our discussion on Montezuma ’ s Revenge because the latter has become a notoriously difficult challenge in RL ; only a handful of techniques ( Bellemare et al . ( 2016 ) ; Horgan et al . ( 2018 ) ; Kulkarni et al . ( 2016 ) ; Ostrovski et al . ( 2017 ) ; Tang et al . ( 2017 ) ) are able to reliably exit the first room ( achieving 400 points ) , in contrast to human players who explore many rooms and achieve a mean score of 4367 points ( Mnih et al . ( 2015 ) ) . In this work , we translate the general idea of Curiosity Search ( “ do something new ” ( Stanton & Clune ( 2016 ) ) ) into a reward for agents that “ go somewhere new ” in the game world . To do so , we divide the game world of Atari into a curiosity grid , each tile of which provides a reward when visited by the agent ( Fig . 1 ) . We then obtain the agent ’ s position from game RAM to indicate where on the grid an agent has visited ; if the game has multiple rooms , as in Montezuma ’ s Revenge , a distinct grid is created for each room . This approach is similar to the bitmasking of states performed in ( Kulkarni et al . ( 2016 ) ) with the main exception that we did not manually specify any objects of interest ( e.g . the key in Montezuma ’ s Revenge ) ; the grid is constructed from equally sized square tiles given an arbitrary grid size , discussed below . While we would ultimately prefer that agents learn to remember where they have been automatically from pixel input , extracting this information from RAM is simpler and allows us to directly examine whether encouraging intra-life novelty can help solve difficult exploration problems in RL . In future work , we could encourage the agent to learn directly from pixels where it has been ( or , more generally , what it has done ) e.g . with recurrent neural networks trained with an auxiliary task like outputting the curiosity grid per time-step . If one did not have access to RAM states , position could be inferred via object detection ( Fragkiadaki et al . ( 2015 ) ) or by implicit state-space representations such as Skip Context Tree Switching ( Bellemare et al . ( 2014 ) ) , the latter of which was shown to work well in a roughly similar counting-based model ( Bellemare et al . ( 2016 ) ) . Also interesting is investigating under what conditions such memories of past behaviors might emerge on their own because they are helpful for exploration ( e.g . in meta-learning ( Duan et al . ( 2018 ) ) ) . Agents were trained via A2C ( Mnih et al . ( 2016 ) ) using the default hyperparameters in OpenAI Baselines ( Dhariwal et al . ( 2017 ) ) . The clipped game reward at any state Rs was replaced with R̂s = βRs + ( 1 − β ) Is , where Is is the intrinsic reward ( 1 if an unexplored tile was just touched and 0 otherwise ) , Rs is the regular game reward clipped to [ −1 , 1 ] ( Mnih et al . ( 2015 ) ) , and β is a constant in the range [ 0 , 1 ] that controls the relative weight of intrinsic and regular game rewards . In the untuned case ( where both rewards are equal ) R̂s = clip ( Rs + Is ) . We chose β = 0.25 after a coarse hyperparameter sweep in Montezuma ’ s Revenge and used untuned values in all other games . Although we treat exploration as binary in this work , Is could also be reduced gradually in proportion to how many times a tile has been visited , similar to the mechanisms in other non-binary count-based approaches ( Bellemare et al . ( 2016 ) ; Brafman & Tennenholtz ( 2001 ) ; Kearns & Singh ( 2002 ) ; Ostrovski et al . ( 2017 ) ) . The intra-life novelty compass of Curiosity Search is traditionally a vector indicating the direction of the nearest rewards ( Stanton & Clune ( 2016 ) ) . Because direction is more appropriate for the first person view of a driving robot ( Lehman & Stanley ( 2011 ) ; Stanton & Clune ( 2016 ) ) and less so for Atari games ( in which the agent sees the entire “ world ” at once ) , we instead use the curiosity grid as an intra-life novelty compass , providing it as an extra network input . The agent must learn how to use this “ map ” to direct itself towards unexplored areas of the game . The curiosity grid is constructed using a per-tile grid size of 16x16 for most games , as described in SI Sec . S1 . To ensure dimensionality equal to normal game state inputs , the grid is resized to match the 84x84 downsampled game screen ( Mnih et al . ( 2015 ) ) . As discussed below , we found that this extra input was helpful , but not necessary to benefit DeepCS on hard RL tasks ; agents perform well if rewarded for visiting new areas even when they do not see where they have ( or have not ) already been ( Fig . 5 ) . All algorithms were trained for 160 million training steps with a frame skip of 4 ( Mnih et al . ( 2015 ) ) , resulting in 640 million game frames per run . 25 runs were performed for each experiment . Plotted results show medians ( solid lines ) , interquartile range ( shaded areas ) , and minimum/maximum scores ( dotted lines ) . We compare DeepCS ( with an underlying A2C implementation ) against A2C alone ; the latter is a synchronous and equally-performing implementation of A3C ( Mnih et al . ( 2016 ) ) . Our A2C algorithm uses 16 actors exploring in parallel ; training curves were obtained by examining the most recent episode score of each actor and plotting the averaged result . Statistical comparisons were obtained via a two-tailed Mann-Whitney U test , which does not assume normality . Videos of end-ofrun agents , open source code , and plot data can be obtained from : keptanonymousforreview . As with any exploration method , encouraging intra-life novelty has possible disadvantages . In a maze environment , Curiosity Search will reward agents for exploring every dead end , even though such states are likely useless for solving the maze itself . Our hypothesis is not that the intra-life exploration of Curiosity Search is superior to that of across-training novelty , but rather that the former is a valuable and interesting tool for solving challenging exploration problems in RL . We also note that while DeepCS might encourage inefficient exploration , ( 1 ) this may not prove problematic as the algorithm still values real game rewards ( how much is a hyperparameter choice ) and may find higher ultimate value in ignoring some dead ends , and ( 2 ) one could use the exploration produced by DeepCS to later produce a distilled , exploitive , efficient policy ( e.g . by annealing the weight on intrinsic rewards at the end of training or filling an experience replay buffer with states observed via DeepCS agents and then training a DQN agent off-policy to only maximize real game rewards ) .

Summary: The authors look at the problem of exploration in deep RL. They propose a “curiosity grid” which is a virtual grid laid out on top of the current level/area that an Atari agent is in. Once an agent enters a new cell of the grid, it obtains a small reward, encouraging the agent to explore all parts of the game. The grid is reset (meaning new rewards can be obtained) after every roll out (meaning the Atari agent has used up all its lives and the game restarts).