Deep exploration by novelty-pursuit with maximum state entropy

1 INTRODUCTION . Efficient exploration is important to learn a ( near- ) optimal policy for reinforcement learning ( RL ) in huge state space ( Sutton & Barto , 1998 ) . Dithering strategies like epsilon-greedy , Gaussian action noise , and Boltzmann exploration are inefficient and require exponential interactions to explore the whole state space . In contrast , deep exploration ( Osband et al. , 2016 ) overcomes this dilemma via temporally extended behaviors with a long-term vision . Recently , proposed methods include the intrinsically motivated goal exploration process ( IMGEP ) ( Forestier et al. , 2017 ) , and maximum state entropy exploration ( MSEE ) ( Hazan et al. , 2019 ) . In particular , IMGEP selects interesting states from the experience buffer as goals for a goal-conditioned exploration policy . In this way , exploration behaviors are naturally temporally-extended via accomplishing self-generated goals . On the other hand , MSEE aims to search for a policy such that it maximizes the entropy of state distribution . In this way , the agent can escape from the local optimum caused by insufficient state exploration . In this paper , we show that the target of maximizing the support of state distribution ( discovering new states ) and maximizing the entropy of state distribution ( unifying visited state distribution ) can be both achieved by the goal-conditioned policy . From this connection , we propose an exploration method called novelty-pursuit . Abstractly , our method performs in two stages : first , it selects a visited state with the least visitation counts as the goal to reach the boundary of the explored region ; then , it takes random actions to explore the non-explored region . An illustration can be seen in Figure 1 . Intuitively , this process is efficient since the agent avoids exploring within the explored region . Besides , the exploration boundary will be expanded further as more and more new states are discovered . Finally , the agent will probably explore the whole state space to find the optimal policy . A naive implementation of the above strategies can lead to inefficient exploration and exploitation on complex environments . First , to tackle the problem of the curse of dimension and exhaustive storage when selecting the least visited states , we approximate the visitation counts via prediction errors given by Random Network Distillation ( Burda et al. , 2019b ) . Besides , we observe that previous methods used in IMGEP ( Forestier et al. , 2017 ) are inefficient to train the goal-conditioned exploration policy . We employ training techniques based on reward shaping ( Ng et al. , 1999 ) and HER ( Andrychowicz et al. , 2017 ) to accelerate training the goal-conditioned policy . Finally , we additionally train an unconditioned exploitation policy to utilize samples collected by the goal-conditioned exploration policy with environment-specific rewards . Thus , the exploration and exploitation is decoupled in our method . Our contributions are summarized as follows : ( 1 ) We disclose that goal-conditioned behaviors can also maximize the state entropy , which bridges the intrinsically motivated goal exploration process and the maximum state entropy explore . ( 2 ) We propose a method called novelty-pursuit from this connection and give practical implementations . ( 3 ) We demonstrate the exploration efficiency of the proposed method and achieve better performance on environments from the maze , Mujoco tasks , to long-horizon video games of SuperMarioBros . 2 BACKGROUND . Reinforcement Learning . In the standard reinforcement learning framework ( Sutton & Barto , 1998 ) a learning agent interacts with a Markov Decision Process ( MDP ) . The sequential decision process is characterized as follows : at each time t , the agent receives a state st from the environment and selects an action at from its policy π ( s , a ) = Pr { a = at|s = st } ; that decision is sent back to the environment , and the environment gives a reward signal r ( st , at ) and transits to the next state st+1 based on the state transition probability pass′ = Pr { s′ = st+1|s = st , a = at } . This process repeats until the agent encounters a terminal state after which the process restarts . The main target of reinforcement learning is to maximize the expected discounted return Eπ [ ∑∞ t=0 γ trt ] in an unknown environment , where γ ∈ ( 0 , 1 ] is a factor that balances the importance of future reward . Without information about environment dynamics and task-specific rewards in advance , the agent needs exploration to discover potential valuable states . Apparently , the learned policy may be sub-optimal if the exploration strategy can not lead to explore the whole state space . Intrinsically Motivated Goal Exploration Process . Intrinsically motivated goal exploration process ( IMGEP ) ( Baranes & Oudeyer , 2009 ; Forestier et al. , 2017 ) relies on a goal-conditioned ( or goal-parameterized ) policy πg for unsupervised exploration . It involves the following steps : 1 ) selecting an intrinsic or interesting state from the experience buffer as the desired goal ; 2 ) exploring with a goal-conditioned policy πg ( s , a , g ) = Pr { at = a|st = s , gt = g } ; 3 ) reusing experience for an exploitaion policy πe ( s , a ) = Pr { at = a|st = s } to maximize the external reward . Note that the performance of exploitation policy πe relies on samples collected by the goal-exploration policy πg . Thus , the criterion of goal selection is crucial for IMGEP . Maximum State Entropy Exploration . Maximum state entropy exploration ( Hazan et al. , 2019 ) aims to search an exploration policy π∗ such that it maximizes the entropy of induced state distribution ( or minimizes the KL-divergence between the uniform distribution and induced state distribution ) among the class of stationary policies ( i.e. , π∗ ∈ argmaxπ H [ dπ ] , where dπ is the state distribution induced by π ) . Without any information about tasks given by the environment , we think maximum state entropy exploration is safe for exploitation . 3 IMGEP WITH MAXIMUM STATE ENTROPY EXPLORATION . In this section , we bridge the intrinsically motivated goal exploration process and maximum state entropy exploration . We begin with practical considerations when maximizing state entropy , then analyze the exploration characteristics of the proposed goal-selection method for IMGEP . In practice , an exact density estimator for high-dimension state space is intractable , and the state space is unknown , which leads to an empirical state distribution over visited states . The differences are important . For example , directly optimizing the entropy of empirical state distribution over visited states is not what we want , because it ignores the non-visited states outside of the empirical state distribution ( see the top row in Fig 2 ) . Instead , we need to first maximize the support of induced state distribution ( i.e. , discovering new states ) , then we maximize the entropy of induced state distribution with full support ( see the bottom row in Fig 2 ) . In the following , we demonstrate that selecting the states with the least visitation counts among visited states as goals can achieve the above functions under some assumptions . Let the set { 1 , 2 , · · · , |S| } denotes the state space S , π1 : t denotes the set of policies { π1 , π2 , · · · , πt } over previous iterations , πt+1 denotes the policy of next iteration , xit denotes the cumulative visitation counts of state i induced by history policies π1 : t , and Nt = ∑|S| i=1 x i t denotes the sum of all state visitation counts . Hence , the entropy of empirical state distribution induced by policies π1 : t is defined as H [ dπ1 : t ( s ) ] = ∑|S| i=1 xit Nt log xit Nt ( Ht for short ) , and the counting measure of empirical state distribution support induced by policies π1 : t is defined as µ [ dπ1 : t ( s ) ] = ∑|S| i=1 I ( xit ≥ 1 ) ( µt for short ) , where I is the indicator function . The theoretical analysis starts with the situation that each iteration the goal-conditioned exploration policy can only select a state to visit without consideration of trajectories towards the goal . Our question is which state to visit gives the most benefits in terms of maximum state entropy . This question is closely related to the goal generation in IMGEP . To facilitate the analysis , let the unit vector e = [ 0 , · · · , 1 , . . . ] ∈ R|S| denotes a choice ( i.e. , e ( i ) = 1 indicates that the policy selects i-th state to visit ) . Note that xt+1 = xt + et with this assumption . Proposition ( Max Counting Measure of Support ) For any state i ∈ { 1 , · · · , |S| } with xit ≥ 0 , unless the unvisited state sets K = { i|xit = 0 } is an empty set , for any choice et such that et ( i ) = 1 with xit = 0 , we have µt+1 = µt + 1 . This Proposition states visiting the non-visited states is to maximize the counting measure of induced state distribution support . The agent improves its policy by discovering new valuable states . In practical applications , we don ’ t have access to non-visited states in advance . In other words , we can ’ t select these non-visited states as goals since they are not contained in the experience buffer . To deal with this problem , we assume that the chance of discovering non-visited states is high when the agent perform random actions to explore around the exploration boundary . The exploration boundary can be understood as the set of visited states with the least visitation counts ( See Figure 1 for the illustration ) . Our assumption is based on the fact that the total visitations counts of the visited region are large and the total visitation counts of the non-visited region are small . In conclusion , the goal-conditioned exploration policy is asked to reach the exploration boundary , then it performs random actions to discover new states to maximize the counting measure . Theorem 1 ( Max Entropy ) For any state i ∈ { 1 , · · · , |S| } with xit ≥ 1 ; for any choice e∗t such that e∗t ( i ) = 1 with i ∈ argminj x j t , we have e ∗ t ∈ argmaxet Ht+1 . We provide the proof in the appendix A.1 . Theorem 1 characterizes the behavior of visiting the states with the least visitations when the whole state space has been explored ( i.e. , the stage after maximizing the counting measure of induced state distribution support ) . Since Theorem 1 still suggests selecting states with the least visitation counts as goals , the above method can also be applied to maximize the entropy of induced state distribution . Actually , it is easy to unify the two stages via a smoothed entropy Hσ ( dπ ) = −Edπ [ log ( dπ ) +σ ] ( Hazan et al. , 2019 ) . For our problem , the definition of entropy is proper by assigning non-visited states with a “ dummy ” visitation counts between 0 and 1 . In that case , Theorem 1 still holds and suggests firstly selecting these non-visited states and subsequently selecting the states with least visitation counts to maximize the smoothed state entropy . The proposed exploration method is called novelty-pursuit . We notice that the above analysis neglects the influences of trajectories towards the exploration boundary . However , the fluctuation of state distribution entropy by the trajectories towards the exploration boundary is less significant from practical considerations . In fact , the goal-conditioned policy should be trained to reach the exploration boundary quickly and pays more efforts to discover new states around the exploration boundary , as our experiment results in Section 5.1 indicate .

Summary: The paper addresses the challenge of intrinsically-driven exploration in tasks with sparse or delayed rewards. First, the authors try to bridge the gap between the objectives of intrinsically-motivated goal generation and maximum state entropy exploration. Then, they propose a new exploration method, called novelty-pursuit, that prescribes the following receipt: first, reach the exploration boundary through a goal-conditioned policy, then take random actions to explore novel states. Finally, the authors compare their approach to a curiosity-driven method based on Random Network Distillation in a wide range of experiments: from toy domains to continuous control, to hard-exploration video games.