NovelD: A Simple yet Effective Exploration Criterion

1 Introduction . Deep reinforcement learning ( RL ) has experienced significant progress over the last several years , with an impressive performance in games like Atari [ 41 , 4 ] , StarCraft [ 61 ] , Go and Chess [ 57–59 ] . However , its success often requires massive computational resources or manually designed dense rewards . The dense rewards are often impractical for real-world settings as they require a significant amount of task-specific domain knowledge . An alternative approach is to allow the agent to explore the environment freely until it reaches the goal . While basic RL exploration criteria ( e.g. , ✏-greedy ) are quite simple , it fails to explore sufficiently in hard tasks . Modern works adopt various Intrinsic Reward ( IR ) designs to guide exploration in hard-exploration settings . However , we observe ( Sec . 4.2 ) in our experiments if there are multiple 1Correspondence to : Tianjun Zhang < tianjunz @ berkeley.edu > . Our code is available at https : //github . com/tianjunz/NovelD . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Start EndRND 1 . RND assigns high IR throughout the environment 2 . RND temporarily focuses on the upper right corner 3 . RND by chance explores the bottom right corner and forgets it already explored upper right 4 . RND re-explores the upper right and forgets the bottom right , gets trapped Low IR High IR Relative High IR regions of interest , these methods sometimes quickly are trapped in one area without sufficiently exploring others . This results in poor overall exploration of large state space [ 2 ] . This is also known as detachment problem in Go-Explore [ 17 ] . On the other hand , provably optimal theoretical RL equipped with optimistic initialization and visitation count bonus explores the environment much more uniformly [ 31 ] . Inspired by that , we introduce a very simple but effective exploration criterion by weighting each novel area approximate equally . Our algorithm uses regulated Novelty Difference ( NovelD ) of consecutive states in a trajectory . The novelty of a state is calculated by Random Network Distillation ( RND [ 11 ] ) . The underlying intuition is that this criterion provides a large intrinsic reward at the boundary between the explored and the unexplored regions ( Fig . 1 ) . As a result , it induces a very different exploration pattern comparing to count-based approaches [ 6 , 10 , 11 , 48 , 5 ] and yields a much broader coverage over the state space . We evaluate NovelD on three very challenging exploration environments : MiniGrid [ 13 ] , NetHack [ 34 ] and Atari games [ 9 ] . MiniGrid is a popular benchmark for evaluating exploration algorithms [ 52 , 12 , 24 ] ; NetHack , built from a real game , is a much more realistic , PG-generated environment with complex goals and skills . Atari games is a widely used benchmark for RL algorithms [ 40 , 11 , 18 ] . NovelD manages to solve all the static environments in MiniGrid within 120M environment steps , without curriculum learning . In contrast , previous SOTA AMIGo [ 12 ] solves 50 % of the tasks , categorized as “ easy ” and “ medium ” , by training a separate goal-generating teacher network in 500M steps . In NetHack , NovelD also outperforms all baselines with a significant margin on various tasks . NovelD is also tested in various Atari games ( e.g. , MonteZuma ’ s Revenge , Venture ) , using image-based input and outperforms RND for both CNN and RNN-based networks . Compared to previous works ( e.g. , RIDE [ 52 ] , AMIGo [ 12 ] and Go-Explore [ 17 ] ) , NovelD has a few design advantages : ( 1 ) in NovelD , there is almost no hyperparameters ; ( 2 ) NovelD is one-stage approach and can be readily combined with any policy learning methods ( e.g. , PPO ) , while many approaches ( e.g. , RIDE , Go-Explore ) are two-stage approaches . ( 3 ) NovelD is asymptotic consistent : its IR vanishes after sufficient exploration , while approaches like RIDE and AMIGo do not . During our experiments , we observe that compared to the count-based approach and RND , NovelD prioritizes the unexplored boundary states , yielding much more efficient and broader exploration patterns . 2 Background . In a Markov Decision Process ( MDP ) , we define state space S , action space A , and ( non-deterministic ) transition function T : S ⇥A ! P ( S ) where P ( S ) is the probability of next state given the current state and action . The goal is to maximize the expected reward R = E [ PT k=0 krt+k=1 ] where rt is the reward and is the discount factor . In this paper , the total reward received at time step t is given by rt = ret + ↵rit , where ret is the extrinsic reward given by the environment , rit is the intrinsic reward from the exploration criterion , and ↵ is a scaling hyperparameter . 3 Exploration via Novelty Difference . 3.1 NovelD Intrinsic Reward Criterion . NovelD is a meta-criterion that can be applied on top of any novelty measure novelty ( s ) that tells how familiar a state s is for the agent . Formally , along a trajectory , we give agent IR if the previous state st has been sufficiently explored but st+1 is not : ri ( st , at , st+1 ) = max [ novelty ( st+1 ) ↵ · novelty ( st ) , 0 ] , ( 1 ) Here ↵ is a scaling factor . Intuitively , if we define { s : novelty ( s ) m } to be an explored region , then NovelD gives intrinsic reward ( IR ) to the agent when it explores beyond the boundary of explored regions . Note that IR is clipped to avoid negative IR if the agent transits back from a novel state to a familiar state . From the equation , only crossing the frontier matters to the intrinsic reward ; if both st+1 and st are novel or familiar , their difference would be small . For each trajectory going towards the frontier/boundary , NovelD assigns an approximately equal IR , regardless of the length ( see Sec . 3.2 ) . Like RIDE [ 52 ] , in our actual implementation , partial observation ot are used instead of the true state st , when st is not available . Episodic Restriction on Intrinsic Reward ( ERIR ) . Simply using Eqn . 1 to guide exploration can result in the agent going back and forth between novel states st+1 and their previous states st. RIDE [ 52 ] avoids this by scaling the intrinsic reward ri ( s ) by the inverse of the episodic state visitation counts . NovelD puts a more aggressive restriction : the agent is only rewarded when it visits the state s for the first time in an episode . Thus , the intrinsic reward of NovelD becomes : ri ( st , at , st+1 ) = max [ novelty ( st+1 ) ↵ · novelty ( st ) , 0 ] ⇤ I { Ne ( st+1 ) = 1 } ( 2 ) Ne here stands for episodic state count and is reset every episode . In contrast , the novelty measure novelty ( · ) is a life-long counter throughout training . Novelty Measure using Random Network Distillation . How to accurate measure the novelty of a state s in large-scale stochastic environments remains an open problem [ 6 ] . Similar to RND [ 11 ] , we use the difference between a random fixed target network and a trainable predictor network 0w as a novelty measure : novelty ( st ) = novelty ( st ; w ) : = k ( st ) 0 w ( st ) k2 , ( 3 ) Once the novelty score for state st is computed , we minimize novelty ( st ; w ) with respect to w and perform a one-step weight update for w , which is the parameter of the predictor network 0w . Therefore , when st is visited again , the quantity novelty ( st ; w ) will be lower , showing that the state has been seen in the past . 3.2 Conceptual Advantages of NovelD over Existing Criteria . We show the distinct preference of exploration directions between RND and NovelD . We also see that NovelD is a consistent algorithm , the intrinsic reward converges to zero after sufficient exploration . Explore via Novelty Difference . Fig . 2 shows the conceptual comparison between NovelD ( Eq . 2 ) and exploration with novelty measure alone ( e.g. , RND ) . We could clearly see their distinctive preferences : our criterion tends to reward more the trajectories that move out of the explored regions ( Trajectory B ) , which typically show a profile of stable novelty measure until a sudden boost when it ventures into an unknown region . On the other hand , count-based approaches can be trapped within explored regions where the trajectories have low but consistent novelty rewards ( Trajectory A ) . : Unsolved Asymptotic Inconsistency . Approaches that define IR as the difference between state representations k ( st ) ( st+1 ) k ( is a learned embedding network ) [ 63 , 38 ] suffer from asymptotic inconsistency . In other words , their IR does not vanish even after sufficient exploration : ri 6 ! 0 when N ! 1 . This is because when the embedding network converges after sufficient exploration , the agent can always obtain non-zero IR if a significant change occurs in state representation ( e.g. , opening a door or picking up a key in MiniGrid ) . Therefore , the learned policy does not maximize the extrinsic reward re , deviating from the goal of RL . Automatic curriculum approaches [ 12 ] ) have similar issues due to an ever-present IR . In contrast , our Eq . 4 is asymptotically consistent as ri ! 0 when N ! 1 . 4 Experiments . We evaluate NovelD on challenging procedurally-generated environment MiniGrid [ 13 ] , several hard-exploration scenarios in Atari games , and NetHack [ 34 ] with sparse rewards ( i.e. , get reward only when reaching the final goal ) . For all the experiments and all exploration approaches ( including NovelD ) , we use PPO [ 55 ] as the base RL algorithm and add intrinsic reward specified by various methods , to encourage exploration . In MiniGrid , we compare NovelD with RND [ 11 ] , ICM [ 49 ] , RIDE [ 52 ] and AMIGo [ 12 ] . We only evaluate AMIGo for 120M steps in our experiments . The algorithm obtains better results when trained for 500M steps as shown in [ 12 ] . For all other baselines , we follow the same training paradigm from [ 52 ] . Mean and standard deviation across four runs of different seeds are computed . NovelD solves all the static tasks provided by MiniGrid . In contrast , all the baselines end up with zero rewards on half of the tasks we tested . In NetHack , NovelD achieves SOTA with a large margin over baselines ( IMPALA [ 19 ] without exploration bonus and RND ) . 4.1 MiniGrid Environments . We mainly use three challenging environments from MiniGird : Multi-Room ( MR ) , Key Corridor ( KC ) and Obstructed Maze ( OM ) . We use these abbreviations for the rest of the paper ( e.g. , OM2Dlh stands for ObstructedMaze2Dlh ) . Fig . 3 shows one example of a rendering on OMFull as well as all the environments we tested with their relative difficulty . In MiniGrid , all environments are of size K⇥K ( K is environment-specific ) where each tile contains an object : wall , door , key , ball , chest , etc . The action space is defined as turn left , turn right , move forward , pick up an object , drop an object , and toggle an object ( e.g. , open or close a door ) . MR consists of a series of rooms connected by doors and the agent must open the door to get to the next room . Success is achieved when the agent reaches the goal . In KC , the agent has to explore the environment to find the key and open the door along the way to achieve success . OM is the hardest : the doors are locked , the keys are hidden in boxes , and doors are obstructed by balls . Results . NovelD manages to solve all the static environments in MiniGrid . In contrast , all baselines solve only up to medium-level tasks and fail to make any progress on more difficult ones . Note that some medium-level tasks we define here are categorized as hard tasks in RIDE and AMIGo ( e.g. , KCS4R3 is labeled as “ KCHard ” and KCS5R3 is labeled as “ KCHarder ” in their works ) . Fig . 4 shows the results of our experiments . Half of the environments ( e.g. , KCS6R3 , OM1Q ) are extremely hard and all the baselines fail . In contrast , NovelD easily solves all listed above without any curriculum learning . We also provide the testing performance for NovelD and all baselines in Tab . 3 . The results are averaged across four seeds and 32 random initialized environments . Table 1 : Visitation counts for the toy corridor environment after 3K episodes . NovelD explores corridors more uniformly than count-based approaches . C1 C2 C3 C4 Entropy Length 40 10 30 10 – Count-Based 66K ± 28K 8K ± 8K 23K ± 35K 13K ± 18K 1.06 ± 0.39 NovelD Tabular 26K ± 2K 28K ± 8K 25K ± 6K 29K ± 9K 1.97 ± 0.02 RND 0.2K ± 0.2K 70K ± 53K 0.2K ± 0.07K 26K ± 44K 0.24 ± 0.28 NovelD 27K ± 6K 23K ± 3K 31K ± 12K 26K ± 8K 1.96 ± 0.05 Table 2 : Entropy of the visitation counts of each room . Such state distribution of NovelD is much more uniform than RND . 0.2M 0.5M 2.0M 5.0M Room1 3.48 / 3.54 3.41 / 3.53 3.51 / 3.56 3.49 / 3.56 Room2 2.87 / 3.09 / 3.23 3.51 / 3.53 3.35 / 3.56 Room3 / / / 4.02 3.42 / 4.01 Room4 / / / 2.74 2.85 / 2.87 ⇤ Results are presented in the order of “ RND / NovelD ” . Multi-Room environments are relatively easy in MiniGrid . However , all the baselines except RIDE fail . As we increase the room size and number ( e.g. , MRN12S10 ) , NovelD can achieve the goal quicker than RIDE . Our method quickly solves these environments within 20M environment steps . On Key Corridor environments , RND , AMIGo , RIDE , IMPALA and NovelD successfully solves KCS3R3 while ICM makes reasonable progress . However , when we increase the room size ( e.g. , KCS5R3 ) , none of the baseline methods work . NovelD manages to solve these environments in 40M environment steps . The agent demonstrates the ability to explore the room and finds the corresponding key to open the door in a randomized , procedurally generated environment . Obstructed Maze environments are also tricky . As shown in Fig . 4 , RIDE and RND manage to solve the easiest task OM2Dlh which doesn ’ t contain any obstructions . In contrast , NovelD not only rapidly solves OM2Dlh , but also solves four more challenging environments including OMFull . These environments have obstructions blocking the door ( as shown in Fig . 3 ) and are much larger than OM2Dlh . Our agent learns to move the obstruction away from the door to open the door and enter the next room in these environments . This “ skill ” is hard to learn since no extrinsic reward is assigned to moving the obstruction . However , learning the skill is critical to achieving the goal .

Summary: The paper proposes a novel exploration bonus for reinforcement learning, NovelD, which consists in adding a weighted clipped difference of the novelty of two consecutive states to the reward. The paper argues that this provides a breadth-first kind of exploration, in contrast to pure novelty based methods that would provide a more depth-first exploration. The paper provides extensive experimental validations, on a variety of environments.