Skew-Explore: Learn faster in continuous spaces with sparse rewards

1 INTRODUCTION Reinforcement Learning ( RL ) is based on performing exploratory actions in a trial-and-error manner and reinforcing those actions that result in superior reward outcomes . Exploration plays an important role in solving a given sequential decision-making problem . A RL agent can not improve its behaviour without receiving rewards exceeding the expectation of the agent , and this happens only as the consequence of properly exploring the environment . In this paper , we propose a method to train a policy which efficiently explores a continuous state space . Our method is particularly well-suited to solve sequential decisionmaking tasks with sparse terminal rewards , i.e. , rewards received at the end of a successful interaction with the environment . We propose to directly maximize the entropy of the history states by exploiting the mutual information between the history states and a number of reference states . To achieve this , we introduce a novel reward function which , given the references , shapes the distribution of the history states . This reward function , combined with goal proposing learning frameworks , maximizes the entropy of the history states . We demonstrate that this way of directly maximizing the state entropy , compared to indirectly maximizing the mutual information ( WardeFarley et al. , 2018 ; Pong et al. , 2019 ) improves the ex- ploration of the state space as well as the convergence speed at solving tasks with sparse terminal rewards . Maximizing the mutual information between the visited states and the goal states , I ( S ; G ) , results in a natural exploration of the environment while learning to reach to different goal states ( Warde-Farley et al. , 2018 ; Pong et al. , 2019 ) . The mutual information can be written as I ( S ; G ) = h ( G ) − h ( G|S ) ; therefore maximizing the mutual information is equivalent to maximizing the en- tropy of the goal state while reducing the conditional entropy ( conditioned on the goal state ) . The first term , encourages the agent to choose its own goal states as diverse as possible , therefore improving the exploration , and the second term forces the agent to reach the different goals it has specified for itself , i.e. , training a goal-conditioned policy , π ( .|s , g ) . Instead of maximizing the mutual information , we propose to maximize the entropy of the visited states directly , i.e. , maximizing h ( S ) = h ( Z ) +h ( S|Z ) −h ( Z|S ) , where Z is a random variable that represents the reference points of promising areas for exploration . Therefore , in our formulation , we have an extra term , h ( S|Z ) , which encourages maximizing the entropy of the state conditioned on the reference points . This extra term , implemented by the proposed reward function , helps the agent to explore better at the vicinity of the references . We call our method Skew-Explore , since similar to Skew-Fit introduced by Pong et al . ( 2019 ) , it skews the distribution of the references toward the less visited states , but instead of directly reaching the goals , it explores the surrounding areas of them . We experimentally demonstrate that the new reward function enables an agent to explore the state space more efficiently in terms of covering larger areas in less time compared to the earlier methods . Furthermore , we demonstrate that our RL agent is capable of solving long-term sequential decision-making problems with sparse rewards faster . We apply the method to three simulated tasks , including a problem to find a trajectory of a YuMi robot end-effector , to open a door of a box , pressing a button inside the box and closing the door . In this case , the sparse reward is given only when the button is pressed and the door is closed , i.e. , at the end of about one minute of continuous interaction with the environment . To validate appropriateness of the trajectory found in simulation , we deployed it on a real YuMi robot , as shown in Figure 1 . The main contributions of this paper can be summarized as ( 1 ) introducing a novel reward function which increases the entropy of the history states much faster compared to the prior work , and ( 2 ) experimentally demonstrating the superiority of the proposed algorithm to solve three different sparse reward sequential decision-making problems . 2 RELATED WORK . Prior works have studied different algorithms for addressing the exploration problem . In this section , we summarize related works in the domain where rewards from the environment are sparse or absent . Intrinsic Reward : One way to encourage exploration is to define an intrinsically-motivated reward , including methods that assimilate the definition of curiosity in psychology ( Oudeyer et al. , 2007 ; Pathak et al. , 2017 ) . These methods have found success in domains like video games ( Ostrovski et al. , 2017 ; Burda et al. , 2018 ) . In these approaches , the ” novelty ” , ” curiosity ” or ” surprise ” of a state is computed as an intrinsic reward using mechanisms such as state-visiting count and prediction error ( Schmidhuber , 1991 ; Stadie et al. , 2015 ; Achiam & Sastry , 2017 ; Pathak et al. , 2017 ) . By considering this information , the agent is encouraged to search for areas that are less visited or have complex dynamics . However , as pointed out by Ecoffet et al . ( 2019 ) , an agent driven by intrinsic reward may suffer from the problem of detaching from the frontiers of high intrinsic reward area . Due to catastrophic forgetting , it may not be able to go back to previous areas that have not yet been fully explored ( Kirkpatrick et al. , 2017 ; Ellefsen et al. , 2015 ) . Our method is able to keep tracking the novelty frontier and train policy to explore different areas in the frontier . Diverse Skill/Option Discovery : Methods that aim to learn a set of behaviours which are distinct from each other , allow the agent to interact with the environment without rewards for a particular task . Gregor et al . ( 2016 ) introduced an option discovery technique based on maximizing the mutual information between the options and the final states of the trajectories . Eysenbach et al . ( 2018 ) ; Florensa et al . ( 2017a ) ; Savinov et al . ( 2018 ) proposed to learn a fixed set of skills by maximizing the mutual information through an internal objective computed using a discriminator . Achiam et al . ( 2018 ) extended the prior works by considering the whole trajectories and introduced a curriculum learning approach that gradually increases the number of skills to be learned . In these works , the exploration is encouraged implicitly through learning diverse skills . However , it is difficult to control the direction of exploration . In our method , we maintain a proposing module which tracks the global information of the states we have visited so far , and keep proposing reference points that guide the agent to the more promising areas for exploration . Self-Goal Proposing : Self-goal proposing methods are often combined with a goal-conditioned policy ( Kaelbling , 1993 ; Andrychowicz et al. , 2017 ) , where a goal ( or task ) generation model is trained jointly with a goal reaching policy . The agent receives rewards in terms of completing the internal tasks which makes it possible to explore the state space without any supervision from the environment . Sukhbaatar et al . ( 2017 ) described a scheme with two agents . The first one proposes tasks by performing a sequence of actions and the other repeats the actions in reverse order . Held et al . ( 2018 ) introduced a method that automatically label and propose goals at the appropriate level of difficulty using adversarial training . Similar works are proposed by Colas et al . ( 2018 ) ; Veeriah et al . ( 2018 ) ; Florensa et al . ( 2017b ) , where goals are selected based on the learning progress . WardeFarley et al . ( 2018 ) trained a goal-conditioned policy by maximizing the mutual information between the goal states and the achieved states . The goals are selected from the agent ’ s recent experience with strategies . Later , Pong et al . ( 2019 ) applied a similar idea of using mutual information . They maximize the entropy of a goal sampling distribution . The focus of these methods is on learning a policy that can reach diverse goals . Although gradually increasing the scale of the goal proposing network , the agent may eventually cover the entire state space , exploration itself is not efficient . In our work , we adopt the same idea of maximizing the entropy of the goal sampling distribution by Pong et al . ( 2019 ) . However , instead of using the goal-conditioned policy , we introduce a reference point-conditioned policy which greatly increases the efficiency of exploration . 3 SKEW-EXPLORE : SEARCHING FOR THE SPARSE REWARD . We discuss the policy learning problem in continuous state and action spaces , which we model as an infinite-horizon Markov decision process ( MDP ) . The MDP is fully characterized by a tuple ( S , A , pa ( s , s′ ) , R′a ( s , s′ ) ) , where S , the state space , and A , the action space , are subsets of Rn , the unknown transition probability p : S × A × S → [ 0 , inf ) indicates the probability density function of the next state s′ given the current state s ∈ S and the action a ∈ A . For each transition , the space associated environment E emits an extrinsic reward according to function R′ : S × A → R. The objective of the agent is to maximize the discounted return , i.e . return R = ∑∞ ts=0 γtsrts , where γ is a discounted factor and rts is the reward received at each step ts . In this study , we consider an agent interacting in an environment E with sparse reward . The sparse reward r is modelled as a truncated Gaussian function with a narrow range . From previous interactions , the agent holds an interaction set It , in which transaction triples ( sj , aj , sj+1 ) , ∀j ∈ { 1 , · · · , T − 1 } are contained . We also extract the states sj from It to form a history state set St , which contains all visited states by the agent until iteration t. The objective of our method is to find an arbitrary external goal in a continuous state space and converge to a policy that maximizes the R as fast as possible . This involves two processes 1 ) Find the external reward through efficient exploration . 2 ) Converge to a policy that maximizes R once the external reward is found . We can use the entropy of the history state set as a neutral objective to encourage exploration , since an agent that maximizes this objective should have visited all valid states uniformly . To describe it mathematically , we define a random variable S to represent the history states that the agent has visited . The distribution of S is estimated from the history state set St. Our goal is to encourage exploration by maximizing the entropy h ( S ) of the history states . However , using the entropy as the intrinsic reward directly may suffer from problems similar to other intrinsic motivated methods ( Schmidhuber , 1991 ; Stadie et al. , 2015 ; Achiam & Sastry , 2017 ; Pathak et al. , 2017 ) . As the reward of the same state is changing , the agent has the risk of detaching from the frontiers of high intrinsic reward area . We introduce a concept called novelty frontier reference point , which can be sampled from a distribution that represents the novelty frontier ( Ecoffet et al. , 2019 ) . The novelty frontier defined in our work represents the areas near the states with lower density in distribution p ( s ) . The frontier reference points are sampled after the distribution of the novelty frontier is updated . We define a Z to represent all the history frontier reference points with probability density p ( z ) estimated from a set Zt that contains all novelty frontier reference points until iteration t. The conditional probability p ( s|z ) defines the behaviour of the agent with respect to each reference point . In this work , we model this behaviour using a state distribution function Kz ( s − z ) parameterized by the displacement between the state and the reference point . The function Kz needs to be chosen carefully as it should satisfy our expectation of the policy behaviour and also , provides an informative reward signal to train the policy . Mathematically , we can rewrite p ( s ) as p ( s ) = ∫ f ( s|z ) p ( z ) dz = ∫ Kz ( s − z ) p ( z ) dz . Generally , Kz ( · ) can be different for different z . However , to reduce the complexity of learning , we constrain Kz ( · ) to be consistent for any z , meaningKz ( s−z ) = K ( s−z ) . The definition ofK ( · ) satisfies the definition of a kernel function . Using K ( s− z ) , p ( s ) can then be further represented as p ( s ) = ∫ K ( s− z ) p ( z ) dz = ( K ∗ p ) ( s ) . By considering the law of convolution of probability distributions , we obtain S = Z + N , where N is a random variable characterized by a density function K ( · ) . Now with this setup , we are able to to analyze our method ’ s performance using information theory . By considering the entropy ’ s relationship with mutual information h ( S ) = h ( S|Z ) +I ( S ; Z ) , we receive the final decomposition of our objective under the novelty frontier reference point-conditioned policy framework h ( S ) = h ( Z ) + h ( S|Z ) − h ( Z|S ) . ( 1 ) Eq . 1 indicates that in order to maximize the h ( S ) , we can individually maximize/minimize each term while making other terms fixed . In the following section , we will explain the optimization process in detail . 3.1 MAXIMIZING h ( Z ) : OBTAINING AN EXPANDING SET OF NOVELTY FRONTIER REFERENCE POINTS As introduced above , h ( Z ) is the entropy estimated from the novelty frontier reference points setZt . To increase h ( Z ) , we need to add a new reference point to Zt such that , the entropy estimated form Zt+1 is larger than the entropy estimated from Zt . In our method , the frontier reference points are sampled from the novelty frontier distribution which represents less history areas according to the current history states . Pong et al . ( 2019 ) proposed a method to skew the distribution of the history states using importance sampling , such that states with lower density can be proposed more often . In our work , we use a similar way to estimate the novelty frontier distribution . There are three steps in our process . In the first step , we estimate the p ( s ) from St using a density estimator e.g . Kernel Density Estimation ( KDE ) . In the second step , we sample Q states { s0 , · · · , sQ } from p ( s ) , and compute the normalized weight for each state using Eq . 2 wi = 1 Yα p ( si ) p ( si ) α α ∈ [ − inf , 0 ) , Yα = N∑ n=1 p ( s = sQ ) p ( s = sQ ) α , ( 2 ) where Yα is a normalizing constant . The state with lower p ( s ) has higher weight and vice versa . Finally , we utilize a generative model training scheme Tg ( · , · ) ( e.g . weighted KDE ) , together with sampled states and weights to get a skewed distribution pskewed ( s ) = Tg ( { s0 , · · · , sn } , { w0 , · · · , wn } ) to represent the novelty frontier distribution . If Q is big enough , by choosing a α appropriately , we are able to expand our frontiers after each iteration . As a consequence , the distribution estimated from Zt will become more and more uniform and its range will become larger and larger , just like annual ring of the tree . The entropy of a continuous uniform function U ( p , q ) is ln ( p − q ) and if the distribution has a larger range , the entropy is larger as well . Fig 2 illustrates the estimated frontier distribution skewed from p ( s ) . 3.2 MAXIMIZING h ( S|Z ) − h ( Z|S ) : INCREASING THE EXPLORATION RANGE AROUND REFERENCE POINTS The conditional entropy of h ( S|Z ) and h ( Z|S ) are highly correlated , maximizing/minimizing them individually are difficult . Therefore , in this section , we consider to maximize h ( S|Z ) - h ( Z|S ) as a whole . Using the relation S = Z + N , we rewrite the expression as h ( S|Z ) − h ( Z|S ) = h ( Z + N|Z ) − h ( Z|Z + N ) , which can be further simplified ( see Appendix D ) as h ( Z|S ) − h ( S|Z ) ≥ h ( N ) − h ( Z ) . This implies that there is a lower bound for the expression h ( S|Z ) − h ( Z|S ) . For a fixed h ( Z ) , we can maximize the lower bound h ( N ) − h ( Z ) by increasing h ( N ) . h ( N ) is related to the shape and variance of the exploration distribution near the reference point . In our method , we model N as a Gaussian distribution with zero mean . In an ideal case , we would like to have as large variance as possible . However , increasing the variance also results in learning difficulty , as we need a longer trajectory to evaluate the performance and more samples to update the network . Therefore , we use the variance to control the trade-off between exploration efficiency and learning efficiency .

Summary: This paper proposes a new exploration algorithm by proposing a new way of generating intrinsic rewards. Specifically, the authors propose to maintain a "novelty frontier" which consists of states that have low-likelihood under some likelihood model trained on their replay buffer. The authors propose to sample from the novelty frontier using a scheme similar to a prior method called Skew-Fit, but replace the VAE with a kernel-based density model. To construct an exploration reward, the authors estimate the KL divergence between the resulting policy state distribution and the desired `state distribution, where the desire state distribution is a Gaussian centered around a point sampled from the novelty frontier.