Deep exploration by novelty-pursuit with maximum state entropy

1 INTRODUCTION . Efficient exploration is important to learn a ( near- ) optimal policy for reinforcement learning ( RL ) in huge state space ( Sutton & Barto , 1998 ) . Dithering strategies like epsilon-greedy , Gaussian action noise , and Boltzmann exploration are inefficient and require exponential interactions to explore the whole state space . In contrast , deep exploration ( Osband et al. , 2016 ) overcomes this dilemma via temporally extended behaviors with a long-term vision . Recently , proposed methods include the intrinsically motivated goal exploration process ( IMGEP ) ( Forestier et al. , 2017 ) , and maximum state entropy exploration ( MSEE ) ( Hazan et al. , 2019 ) . In particular , IMGEP selects interesting states from the experience buffer as goals for a goal-conditioned exploration policy . In this way , exploration behaviors are naturally temporally-extended via accomplishing self-generated goals . On the other hand , MSEE aims to search for a policy such that it maximizes the entropy of state distribution . In this way , the agent can escape from the local optimum caused by insufficient state exploration . In this paper , we show that the target of maximizing the support of state distribution ( discovering new states ) and maximizing the entropy of state distribution ( unifying visited state distribution ) can be both achieved by the goal-conditioned policy . From this connection , we propose an exploration method called novelty-pursuit . Abstractly , our method performs in two stages : first , it selects a visited state with the least visitation counts as the goal to reach the boundary of the explored region ; then , it takes random actions to explore the non-explored region . An illustration can be seen in Figure 1 . Intuitively , this process is efficient since the agent avoids exploring within the explored region . Besides , the exploration boundary will be expanded further as more and more new states are discovered . Finally , the agent will probably explore the whole state space to find the optimal policy . A naive implementation of the above strategies can lead to inefficient exploration and exploitation on complex environments . First , to tackle the problem of the curse of dimension and exhaustive storage when selecting the least visited states , we approximate the visitation counts via prediction errors given by Random Network Distillation ( Burda et al. , 2019b ) . Besides , we observe that previous methods used in IMGEP ( Forestier et al. , 2017 ) are inefficient to train the goal-conditioned exploration policy . We employ training techniques based on reward shaping ( Ng et al. , 1999 ) and HER ( Andrychowicz et al. , 2017 ) to accelerate training the goal-conditioned policy . Finally , we additionally train an unconditioned exploitation policy to utilize samples collected by the goal-conditioned exploration policy with environment-specific rewards . Thus , the exploration and exploitation is decoupled in our method . Our contributions are summarized as follows : ( 1 ) We disclose that goal-conditioned behaviors can also maximize the state entropy , which bridges the intrinsically motivated goal exploration process and the maximum state entropy explore . ( 2 ) We propose a method called novelty-pursuit from this connection and give practical implementations . ( 3 ) We demonstrate the exploration efficiency of the proposed method and achieve better performance on environments from the maze , Mujoco tasks , to long-horizon video games of SuperMarioBros . 2 BACKGROUND . Reinforcement Learning . In the standard reinforcement learning framework ( Sutton & Barto , 1998 ) a learning agent interacts with a Markov Decision Process ( MDP ) . The sequential decision process is characterized as follows : at each time t , the agent receives a state st from the environment and selects an action at from its policy π ( s , a ) = Pr { a = at|s = st } ; that decision is sent back to the environment , and the environment gives a reward signal r ( st , at ) and transits to the next state st+1 based on the state transition probability pass′ = Pr { s′ = st+1|s = st , a = at } . This process repeats until the agent encounters a terminal state after which the process restarts . The main target of reinforcement learning is to maximize the expected discounted return Eπ [ ∑∞ t=0 γ trt ] in an unknown environment , where γ ∈ ( 0 , 1 ] is a factor that balances the importance of future reward . Without information about environment dynamics and task-specific rewards in advance , the agent needs exploration to discover potential valuable states . Apparently , the learned policy may be sub-optimal if the exploration strategy can not lead to explore the whole state space . Intrinsically Motivated Goal Exploration Process . Intrinsically motivated goal exploration process ( IMGEP ) ( Baranes & Oudeyer , 2009 ; Forestier et al. , 2017 ) relies on a goal-conditioned ( or goal-parameterized ) policy πg for unsupervised exploration . It involves the following steps : 1 ) selecting an intrinsic or interesting state from the experience buffer as the desired goal ; 2 ) exploring with a goal-conditioned policy πg ( s , a , g ) = Pr { at = a|st = s , gt = g } ; 3 ) reusing experience for an exploitaion policy πe ( s , a ) = Pr { at = a|st = s } to maximize the external reward . Note that the performance of exploitation policy πe relies on samples collected by the goal-exploration policy πg . Thus , the criterion of goal selection is crucial for IMGEP . Maximum State Entropy Exploration . Maximum state entropy exploration ( Hazan et al. , 2019 ) aims to search an exploration policy π∗ such that it maximizes the entropy of induced state distribution ( or minimizes the KL-divergence between the uniform distribution and induced state distribution ) among the class of stationary policies ( i.e. , π∗ ∈ argmaxπ H [ dπ ] , where dπ is the state distribution induced by π ) . Without any information about tasks given by the environment , we think maximum state entropy exploration is safe for exploitation . 3 IMGEP WITH MAXIMUM STATE ENTROPY EXPLORATION . In this section , we bridge the intrinsically motivated goal exploration process and maximum state entropy exploration . We begin with practical considerations when maximizing state entropy , then analyze the exploration characteristics of the proposed goal-selection method for IMGEP . In practice , an exact density estimator for high-dimension state space is intractable , and the state space is unknown , which leads to an empirical state distribution over visited states . The differences are important . For example , directly optimizing the entropy of empirical state distribution over visited states is not what we want , because it ignores the non-visited states outside of the empirical state distribution ( see the top row in Fig 2 ) . Instead , we need to first maximize the support of induced state distribution ( i.e. , discovering new states ) , then we maximize the entropy of induced state distribution with full support ( see the bottom row in Fig 2 ) . In the following , we demonstrate that selecting the states with the least visitation counts among visited states as goals can achieve the above functions under some assumptions . Let the set { 1 , 2 , · · · , |S| } denotes the state space S , π1 : t denotes the set of policies { π1 , π2 , · · · , πt } over previous iterations , πt+1 denotes the policy of next iteration , xit denotes the cumulative visitation counts of state i induced by history policies π1 : t , and Nt = ∑|S| i=1 x i t denotes the sum of all state visitation counts . Hence , the entropy of empirical state distribution induced by policies π1 : t is defined as H [ dπ1 : t ( s ) ] = ∑|S| i=1 xit Nt log xit Nt ( Ht for short ) , and the counting measure of empirical state distribution support induced by policies π1 : t is defined as µ [ dπ1 : t ( s ) ] = ∑|S| i=1 I ( xit ≥ 1 ) ( µt for short ) , where I is the indicator function . The theoretical analysis starts with the situation that each iteration the goal-conditioned exploration policy can only select a state to visit without consideration of trajectories towards the goal . Our question is which state to visit gives the most benefits in terms of maximum state entropy . This question is closely related to the goal generation in IMGEP . To facilitate the analysis , let the unit vector e = [ 0 , · · · , 1 , . . . ] ∈ R|S| denotes a choice ( i.e. , e ( i ) = 1 indicates that the policy selects i-th state to visit ) . Note that xt+1 = xt + et with this assumption . Proposition ( Max Counting Measure of Support ) For any state i ∈ { 1 , · · · , |S| } with xit ≥ 0 , unless the unvisited state sets K = { i|xit = 0 } is an empty set , for any choice et such that et ( i ) = 1 with xit = 0 , we have µt+1 = µt + 1 . This Proposition states visiting the non-visited states is to maximize the counting measure of induced state distribution support . The agent improves its policy by discovering new valuable states . In practical applications , we don ’ t have access to non-visited states in advance . In other words , we can ’ t select these non-visited states as goals since they are not contained in the experience buffer . To deal with this problem , we assume that the chance of discovering non-visited states is high when the agent perform random actions to explore around the exploration boundary . The exploration boundary can be understood as the set of visited states with the least visitation counts ( See Figure 1 for the illustration ) . Our assumption is based on the fact that the total visitations counts of the visited region are large and the total visitation counts of the non-visited region are small . In conclusion , the goal-conditioned exploration policy is asked to reach the exploration boundary , then it performs random actions to discover new states to maximize the counting measure . Theorem 1 ( Max Entropy ) For any state i ∈ { 1 , · · · , |S| } with xit ≥ 1 ; for any choice e∗t such that e∗t ( i ) = 1 with i ∈ argminj x j t , we have e ∗ t ∈ argmaxet Ht+1 . We provide the proof in the appendix A.1 . Theorem 1 characterizes the behavior of visiting the states with the least visitations when the whole state space has been explored ( i.e. , the stage after maximizing the counting measure of induced state distribution support ) . Since Theorem 1 still suggests selecting states with the least visitation counts as goals , the above method can also be applied to maximize the entropy of induced state distribution . Actually , it is easy to unify the two stages via a smoothed entropy Hσ ( dπ ) = −Edπ [ log ( dπ ) +σ ] ( Hazan et al. , 2019 ) . For our problem , the definition of entropy is proper by assigning non-visited states with a “ dummy ” visitation counts between 0 and 1 . In that case , Theorem 1 still holds and suggests firstly selecting these non-visited states and subsequently selecting the states with least visitation counts to maximize the smoothed state entropy . The proposed exploration method is called novelty-pursuit . We notice that the above analysis neglects the influences of trajectories towards the exploration boundary . However , the fluctuation of state distribution entropy by the trajectories towards the exploration boundary is less significant from practical considerations . In fact , the goal-conditioned policy should be trained to reach the exploration boundary quickly and pays more efforts to discover new states around the exploration boundary , as our experiment results in Section 5.1 indicate .

Summary: This paper proposes novelty-pursuit for exploration in large state space. In theory, novelty-pursuit is motivated by connecting intrinsically motivated goal exploration process (IMGEP) and the maximum state entropy exploration (MSEE), showing that exploring least visited state can increase state distribution entropy most. In practice, novelty-pursuit works in two stages: First, it selects a goal (with largest value prediction error) to train a goal reaching policy to reach the boundary of explored and unexplored states. Second, after reaching goal states, it uses a randomly policy to explore, hopefully can get to unexplored states. Experiments on Empty Room show that the novelty-pursuit with perfect goal reaching policy and visit count information can maximize state distribution entropy. Experiments on Empty Room, Four Rooms, FetchReach and SuperMarioBros show that the proposed method can achieve better performance than vanilla (policy gradient?) and bonus (exploration bonus using Random Network Distillation).