NovelD: A Simple yet Effective Exploration Criterion

1 Introduction . Deep reinforcement learning ( RL ) has experienced significant progress over the last several years , with an impressive performance in games like Atari [ 41 , 4 ] , StarCraft [ 61 ] , Go and Chess [ 57–59 ] . However , its success often requires massive computational resources or manually designed dense rewards . The dense rewards are often impractical for real-world settings as they require a significant amount of task-specific domain knowledge . An alternative approach is to allow the agent to explore the environment freely until it reaches the goal . While basic RL exploration criteria ( e.g. , ✏-greedy ) are quite simple , it fails to explore sufficiently in hard tasks . Modern works adopt various Intrinsic Reward ( IR ) designs to guide exploration in hard-exploration settings . However , we observe ( Sec . 4.2 ) in our experiments if there are multiple 1Correspondence to : Tianjun Zhang < tianjunz @ berkeley.edu > . Our code is available at https : //github . com/tianjunz/NovelD . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Start EndRND 1 . RND assigns high IR throughout the environment 2 . RND temporarily focuses on the upper right corner 3 . RND by chance explores the bottom right corner and forgets it already explored upper right 4 . RND re-explores the upper right and forgets the bottom right , gets trapped Low IR High IR Relative High IR regions of interest , these methods sometimes quickly are trapped in one area without sufficiently exploring others . This results in poor overall exploration of large state space [ 2 ] . This is also known as detachment problem in Go-Explore [ 17 ] . On the other hand , provably optimal theoretical RL equipped with optimistic initialization and visitation count bonus explores the environment much more uniformly [ 31 ] . Inspired by that , we introduce a very simple but effective exploration criterion by weighting each novel area approximate equally . Our algorithm uses regulated Novelty Difference ( NovelD ) of consecutive states in a trajectory . The novelty of a state is calculated by Random Network Distillation ( RND [ 11 ] ) . The underlying intuition is that this criterion provides a large intrinsic reward at the boundary between the explored and the unexplored regions ( Fig . 1 ) . As a result , it induces a very different exploration pattern comparing to count-based approaches [ 6 , 10 , 11 , 48 , 5 ] and yields a much broader coverage over the state space . We evaluate NovelD on three very challenging exploration environments : MiniGrid [ 13 ] , NetHack [ 34 ] and Atari games [ 9 ] . MiniGrid is a popular benchmark for evaluating exploration algorithms [ 52 , 12 , 24 ] ; NetHack , built from a real game , is a much more realistic , PG-generated environment with complex goals and skills . Atari games is a widely used benchmark for RL algorithms [ 40 , 11 , 18 ] . NovelD manages to solve all the static environments in MiniGrid within 120M environment steps , without curriculum learning . In contrast , previous SOTA AMIGo [ 12 ] solves 50 % of the tasks , categorized as “ easy ” and “ medium ” , by training a separate goal-generating teacher network in 500M steps . In NetHack , NovelD also outperforms all baselines with a significant margin on various tasks . NovelD is also tested in various Atari games ( e.g. , MonteZuma ’ s Revenge , Venture ) , using image-based input and outperforms RND for both CNN and RNN-based networks . Compared to previous works ( e.g. , RIDE [ 52 ] , AMIGo [ 12 ] and Go-Explore [ 17 ] ) , NovelD has a few design advantages : ( 1 ) in NovelD , there is almost no hyperparameters ; ( 2 ) NovelD is one-stage approach and can be readily combined with any policy learning methods ( e.g. , PPO ) , while many approaches ( e.g. , RIDE , Go-Explore ) are two-stage approaches . ( 3 ) NovelD is asymptotic consistent : its IR vanishes after sufficient exploration , while approaches like RIDE and AMIGo do not . During our experiments , we observe that compared to the count-based approach and RND , NovelD prioritizes the unexplored boundary states , yielding much more efficient and broader exploration patterns . 2 Background . In a Markov Decision Process ( MDP ) , we define state space S , action space A , and ( non-deterministic ) transition function T : S ⇥A ! P ( S ) where P ( S ) is the probability of next state given the current state and action . The goal is to maximize the expected reward R = E [ PT k=0 krt+k=1 ] where rt is the reward and is the discount factor . In this paper , the total reward received at time step t is given by rt = ret + ↵rit , where ret is the extrinsic reward given by the environment , rit is the intrinsic reward from the exploration criterion , and ↵ is a scaling hyperparameter . 3 Exploration via Novelty Difference . 3.1 NovelD Intrinsic Reward Criterion . NovelD is a meta-criterion that can be applied on top of any novelty measure novelty ( s ) that tells how familiar a state s is for the agent . Formally , along a trajectory , we give agent IR if the previous state st has been sufficiently explored but st+1 is not : ri ( st , at , st+1 ) = max [ novelty ( st+1 ) ↵ · novelty ( st ) , 0 ] , ( 1 ) Here ↵ is a scaling factor . Intuitively , if we define { s : novelty ( s ) m } to be an explored region , then NovelD gives intrinsic reward ( IR ) to the agent when it explores beyond the boundary of explored regions . Note that IR is clipped to avoid negative IR if the agent transits back from a novel state to a familiar state . From the equation , only crossing the frontier matters to the intrinsic reward ; if both st+1 and st are novel or familiar , their difference would be small . For each trajectory going towards the frontier/boundary , NovelD assigns an approximately equal IR , regardless of the length ( see Sec . 3.2 ) . Like RIDE [ 52 ] , in our actual implementation , partial observation ot are used instead of the true state st , when st is not available . Episodic Restriction on Intrinsic Reward ( ERIR ) . Simply using Eqn . 1 to guide exploration can result in the agent going back and forth between novel states st+1 and their previous states st. RIDE [ 52 ] avoids this by scaling the intrinsic reward ri ( s ) by the inverse of the episodic state visitation counts . NovelD puts a more aggressive restriction : the agent is only rewarded when it visits the state s for the first time in an episode . Thus , the intrinsic reward of NovelD becomes : ri ( st , at , st+1 ) = max [ novelty ( st+1 ) ↵ · novelty ( st ) , 0 ] ⇤ I { Ne ( st+1 ) = 1 } ( 2 ) Ne here stands for episodic state count and is reset every episode . In contrast , the novelty measure novelty ( · ) is a life-long counter throughout training . Novelty Measure using Random Network Distillation . How to accurate measure the novelty of a state s in large-scale stochastic environments remains an open problem [ 6 ] . Similar to RND [ 11 ] , we use the difference between a random fixed target network and a trainable predictor network 0w as a novelty measure : novelty ( st ) = novelty ( st ; w ) : = k ( st ) 0 w ( st ) k2 , ( 3 ) Once the novelty score for state st is computed , we minimize novelty ( st ; w ) with respect to w and perform a one-step weight update for w , which is the parameter of the predictor network 0w . Therefore , when st is visited again , the quantity novelty ( st ; w ) will be lower , showing that the state has been seen in the past . 3.2 Conceptual Advantages of NovelD over Existing Criteria . We show the distinct preference of exploration directions between RND and NovelD . We also see that NovelD is a consistent algorithm , the intrinsic reward converges to zero after sufficient exploration . Explore via Novelty Difference . Fig . 2 shows the conceptual comparison between NovelD ( Eq . 2 ) and exploration with novelty measure alone ( e.g. , RND ) . We could clearly see their distinctive preferences : our criterion tends to reward more the trajectories that move out of the explored regions ( Trajectory B ) , which typically show a profile of stable novelty measure until a sudden boost when it ventures into an unknown region . On the other hand , count-based approaches can be trapped within explored regions where the trajectories have low but consistent novelty rewards ( Trajectory A ) . : Unsolved Asymptotic Inconsistency . Approaches that define IR as the difference between state representations k ( st ) ( st+1 ) k ( is a learned embedding network ) [ 63 , 38 ] suffer from asymptotic inconsistency . In other words , their IR does not vanish even after sufficient exploration : ri 6 ! 0 when N ! 1 . This is because when the embedding network converges after sufficient exploration , the agent can always obtain non-zero IR if a significant change occurs in state representation ( e.g. , opening a door or picking up a key in MiniGrid ) . Therefore , the learned policy does not maximize the extrinsic reward re , deviating from the goal of RL . Automatic curriculum approaches [ 12 ] ) have similar issues due to an ever-present IR . In contrast , our Eq . 4 is asymptotically consistent as ri ! 0 when N ! 1 . 4 Experiments . We evaluate NovelD on challenging procedurally-generated environment MiniGrid [ 13 ] , several hard-exploration scenarios in Atari games , and NetHack [ 34 ] with sparse rewards ( i.e. , get reward only when reaching the final goal ) . For all the experiments and all exploration approaches ( including NovelD ) , we use PPO [ 55 ] as the base RL algorithm and add intrinsic reward specified by various methods , to encourage exploration . In MiniGrid , we compare NovelD with RND [ 11 ] , ICM [ 49 ] , RIDE [ 52 ] and AMIGo [ 12 ] . We only evaluate AMIGo for 120M steps in our experiments . The algorithm obtains better results when trained for 500M steps as shown in [ 12 ] . For all other baselines , we follow the same training paradigm from [ 52 ] . Mean and standard deviation across four runs of different seeds are computed . NovelD solves all the static tasks provided by MiniGrid . In contrast , all the baselines end up with zero rewards on half of the tasks we tested . In NetHack , NovelD achieves SOTA with a large margin over baselines ( IMPALA [ 19 ] without exploration bonus and RND ) . 4.1 MiniGrid Environments . We mainly use three challenging environments from MiniGird : Multi-Room ( MR ) , Key Corridor ( KC ) and Obstructed Maze ( OM ) . We use these abbreviations for the rest of the paper ( e.g. , OM2Dlh stands for ObstructedMaze2Dlh ) . Fig . 3 shows one example of a rendering on OMFull as well as all the environments we tested with their relative difficulty . In MiniGrid , all environments are of size K⇥K ( K is environment-specific ) where each tile contains an object : wall , door , key , ball , chest , etc . The action space is defined as turn left , turn right , move forward , pick up an object , drop an object , and toggle an object ( e.g. , open or close a door ) . MR consists of a series of rooms connected by doors and the agent must open the door to get to the next room . Success is achieved when the agent reaches the goal . In KC , the agent has to explore the environment to find the key and open the door along the way to achieve success . OM is the hardest : the doors are locked , the keys are hidden in boxes , and doors are obstructed by balls . Results . NovelD manages to solve all the static environments in MiniGrid . In contrast , all baselines solve only up to medium-level tasks and fail to make any progress on more difficult ones . Note that some medium-level tasks we define here are categorized as hard tasks in RIDE and AMIGo ( e.g. , KCS4R3 is labeled as “ KCHard ” and KCS5R3 is labeled as “ KCHarder ” in their works ) . Fig . 4 shows the results of our experiments . Half of the environments ( e.g. , KCS6R3 , OM1Q ) are extremely hard and all the baselines fail . In contrast , NovelD easily solves all listed above without any curriculum learning . We also provide the testing performance for NovelD and all baselines in Tab . 3 . The results are averaged across four seeds and 32 random initialized environments . Table 1 : Visitation counts for the toy corridor environment after 3K episodes . NovelD explores corridors more uniformly than count-based approaches . C1 C2 C3 C4 Entropy Length 40 10 30 10 – Count-Based 66K ± 28K 8K ± 8K 23K ± 35K 13K ± 18K 1.06 ± 0.39 NovelD Tabular 26K ± 2K 28K ± 8K 25K ± 6K 29K ± 9K 1.97 ± 0.02 RND 0.2K ± 0.2K 70K ± 53K 0.2K ± 0.07K 26K ± 44K 0.24 ± 0.28 NovelD 27K ± 6K 23K ± 3K 31K ± 12K 26K ± 8K 1.96 ± 0.05 Table 2 : Entropy of the visitation counts of each room . Such state distribution of NovelD is much more uniform than RND . 0.2M 0.5M 2.0M 5.0M Room1 3.48 / 3.54 3.41 / 3.53 3.51 / 3.56 3.49 / 3.56 Room2 2.87 / 3.09 / 3.23 3.51 / 3.53 3.35 / 3.56 Room3 / / / 4.02 3.42 / 4.01 Room4 / / / 2.74 2.85 / 2.87 ⇤ Results are presented in the order of “ RND / NovelD ” . Multi-Room environments are relatively easy in MiniGrid . However , all the baselines except RIDE fail . As we increase the room size and number ( e.g. , MRN12S10 ) , NovelD can achieve the goal quicker than RIDE . Our method quickly solves these environments within 20M environment steps . On Key Corridor environments , RND , AMIGo , RIDE , IMPALA and NovelD successfully solves KCS3R3 while ICM makes reasonable progress . However , when we increase the room size ( e.g. , KCS5R3 ) , none of the baseline methods work . NovelD manages to solve these environments in 40M environment steps . The agent demonstrates the ability to explore the room and finds the corresponding key to open the door in a randomized , procedurally generated environment . Obstructed Maze environments are also tricky . As shown in Fig . 4 , RIDE and RND manage to solve the easiest task OM2Dlh which doesn ’ t contain any obstructions . In contrast , NovelD not only rapidly solves OM2Dlh , but also solves four more challenging environments including OMFull . These environments have obstructions blocking the door ( as shown in Fig . 3 ) and are much larger than OM2Dlh . Our agent learns to move the obstruction away from the door to open the door and enter the next room in these environments . This “ skill ” is hard to learn since no extrinsic reward is assigned to moving the obstruction . However , learning the skill is critical to achieving the goal .

Summary: This paper presents a simple exploration criterion NovelD. The basic idea is to use the difference of the novelties between the next state and the current state as an intrinsic reward, which is the "boundary" of the explored and unexplored regions. This idea is implemented with a max operation to avoid negative rewards, a binary indicator that only rewards the state visited for the first time, and RDN as the novelty measure. The experiments are conducted on MiniGrid, NetHack, and some hard exploration environments in Atari.