Model Ensemble-Based Intrinsic Reward for Sparse Reward Reinforcement Learning

In this paper , a new intrinsic reward generation method for sparse-reward reinforcement learning is proposed based on an ensemble of dynamics models . In the proposed method , the mixture of multiple dynamics models is used to approximate the true unknown transition probability , and the intrinsic reward is designed as the minimum of the surprise seen from each dynamics model to the mixture of the dynamics models . In order to show the effectiveness of the proposed intrinsic reward generation method , a working algorithm is constructed by combining the proposed intrinsic reward generation method with the proximal policy optimization ( PPO ) algorithm . Numerical results show that for representative locomotion tasks , the proposed model-ensemble-based intrinsic reward generation method outperforms the previous methods based on a single dynamics model . 1 INTRODUCTION . Reinforcement learning ( RL ) with sparse reward is an active research area ( Andrychowicz et al. , 2017 ; de Abril & Kanai , 2018 ; Kim et al. , 2018 ; Oh et al. , 2018 ; Tang et al. , 2017 ) . In typical model-free RL , an agent learns a policy to maximize the expected cumulative reward under the circumstance that the agent receives a non-zero reward from the environment for each action of the agent . On the contrary , in sparse reward RL , the environment does not return a non-zero reward for every action of the agent but returns a non-zero reward only when certain conditions are met . Such situations are encountered in many action control problems ( Andrychowicz et al. , 2017 ; Houthooft et al. , 2016 ; Oh et al. , 2018 ) . As in conventional RL , exploration is important at the early stage of learning in sparse reward RL , whereas the balance between exploration and exploitation is required on the later stage . Methods such as the -greedy strategy ( Mnih et al. , 2015 ; Van Hasselt et al. , 2016 ) and the control of policy gradient with Gaussian random noise ( Duan et al. , 2016 ; Schulman et al. , 2015a ) have been applied to various tasks for exploration . However , these methods have been revealed to be insufficient for successful learning when reward is sparse ( Achiam & Sastry , 2017 ) . In order to overcome such difficulty , intrinsically motivated RL has been studied to stimulate better exploration by generating intrinsic reward for each action by the agent itself , even when reward is sparse . Recently , many intrinsically-motivated RL algorithms have been devised to deal with the sparsity of reward , e.g. , based on the notion of curiosity ( Houthooft et al. , 2016 ; Pathak et al. , 2017 ) and surprise ( Achiam & Sastry , 2017 ) . It is shown that these algorithms are successful and outperform the previous approaches . In essence , these algorithms use a single estimation model for the next state or the environment dynamics to generate intrinsic reward . In this paper , in order to further improve the performance of sparse reward model-free RL , we propose a new method to generate intrinsic reward based on an ensemble of estimation models for the environment dynamics . The rationale behind our approach is that by using a mixture of several distributions , we can increase degrees of freedom for modeling the unknown underlying model dynamics and designing a better reward from the ensemble of estimation models . Numerical results show that the proposed model-ensemble-based intrinsic reward generation method yields improved performance as compared to existing reward generation methods for continuous control with sparse reward setting . 2 THE PROPOSED METHOD . 2.1 PRELIMINARIES . In this paper , we consider a discrete-time continuous-state Markov Decision Process ( MDP ) , denoted as ( S , A , P , r , ρ0 , γ ) , where S andA are the sets of states and actions , respectively , P : S×A×S → [ 0 , 1 ] is the transition probability function ( called model dynamics ) , r : S×A×S → R is the extrinsic reward function , ρ0 : S → [ 0 , 1 ] is the distribution of the initial state , and γ is the discounting factor . A ( stochastic ) policy is represented by π : S ×A → [ 0 , 1 ] , where π ( a|s ) represents the probability of choosing action a ∈ A for given state s ∈ S. In sparse reward RL , the environment does not return a non-zero reward for every action but returns a non-zero reward only when certain conditions are met by the current state , the action and the next state ( Andrychowicz et al. , 2017 ; Houthooft et al. , 2016 ; Oh et al. , 2018 ) . The goal of this paper is to optimize the policy π to maximize the expected cumulative return η ( π ) by properly generating intrinsic reward in such sparse reward environments . We assume that the true transition model P is unknown to the agent . Intrinsically-motivated RL adds a properly designed intrinsic reward to the actual extrinsic reward to yield a non-zero total reward for training even when the extrinsic reward returned by the environment is zero ( de Abril & Kanai , 2018 ; Pathak et al. , 2017 ; Tang et al. , 2017 ) . One way to design such an intrinsic reward for action control is based on surprise , which is a measure of the unexpectedness of observing the next state for a given current state and action pair and is especially useful to yield better exploration ( Achiam & Sastry , 2017 ) . In this context , a recent work ( Achiam & Sastry , 2017 ) proposed a promising direction of intrinsic reward design in which the agent tries to optimize its policy π according to max π { η ( π ) + c E ( s , a ) ∼π [ DKL ( P ||Pφ ) | ( s , a ) ] } ( 1 ) for some constant c > 0 , where Pφ is the learning model parameterized by φ that the agent has regarding the true unknown transition probability P of the environment . DKL ( P ||Pφ ) | ( s , a ) is the Kullback-Leibler divergence ( KLD ) between two distributions P and Pφ of the next state for given current state-action pair ( s , a ) , and E ( s , a ) ∼π is the expectation over ( s , a ) following the policy π . Thus , the surprise is quantified as E ( s , a ) ∼π [ DKL ( P ||Pφ ) | ( s , a ) ] ( Achiam & Sastry , 2017 ) . Furthermore , the KLD DKL ( P ||Pφ ) | ( st , at ) at timestep t can be lower-bounded as DKL ( P ||Pφ ) | ( st , at ) ≥ E [ log Pφ′ ( ·|st , at ) Pφ ( ·|st , at ) ] ( 2 ) with an arbitrary choice of the parameter φ′ . Therefore , the intrinsic reward at timestep t is determined as r′t , int ( st , at , st+1 ) = log Pφ′ ( st+1|st , at ) Pφ ( st+1|st , at ) ( Achiam & Sastry , 2017 ) , where Pφ′ needs to be designed properly . With φ′ = φ ( t ) and φ = φ ( t− ) , where Pφ ( t ) and Pφ ( t− ) are respectively the agent ’ s model for P at timestep t and the model before the update at timestep t , the intrinsic reward is given by the computable quantity named as the 1-step surprise : r1−stept , int ( st , at , st+1 ) = log Pφ ( t ) ( st+1|st , at ) Pφ ( t− ) ( st+1|st , at ) . ( 3 ) The proposed 1-step intrinsic reward performs well compared to the previously designed intrinsic reward , and it is based on a single model Pφ for P , where Pφ for given ( s , a ) is modeled as Gaussian distribution ( Achiam & Sastry , 2017 ) . 2.2 INTRINSIC REWARD DESIGN FROM AN ENSEMBLE OF DYNAMICS MODELS . In this paper , we take the principle that DKL ( P ||Pφ ) | ( s , a ) is a reasonable measure for surprise to promote exploration , and generalize the intrinsic reward design under this measure . However , instead of using a single learning model for P as in the previous approach , we propose using an ensemble of K dynamics models Pφ1 , · · · , PφK for P , constructing the mixture distribution PK = K∑ i=1 qiPφi ( 4 ) with the mixing coefficients qi ≥ 0 and ∑K i=1 qi = 1 , and using PK in ( 4 ) as an estimate for the true unknown P . The rationale behind this is that by using a mixture of several distributions we increase degrees of freedom for modeling the underlying model dynamics and designing a better intrinsic reward . For the j-th model Pφj , j = 1 , · · · , K , we have DKL ( P ||Pφj ) | ( st , at ) ≥ E [ log PK ( ·|st , at ) Pφj ( ·|st , at ) ] ( 5 ) as in ( 2 ) . Thus , for Pφj , the intrinsic reward at timestep t is determined as r̃jt , int ( st , at , st+1 ) = log PK ( st+1|st , at ) Pφj ( st+1|st , at ) . ( 6 ) Furthermore , ( 6 ) can be modified to yield a 1-step surprise intrinsic reward as rjt , int ( st , at , st+1 ) = log ∑K i=1 qiPφil ( t ) ( st+1|st , at ) Pφj l ( t ) −1 ( st+1|st , at ) ( 7 ) where Pφj l ( t ) and Pφj l ( t ) −1 are the j-th model at the update period l corresponding to timestep t and the previous update period l − 1 , respectively ( l ( t ) will become clear in the subsection 2.3 ) . Since the mixture model ( 4 ) has the increased model order for modeling the underlying dynamics distribution beyond single-mode distributions , we have more freedom to design intrinsic reward . That is , we now have K values , rjt , int ( st , at , st+1 ) , j = 1 , · · · , K , for candidates for intrinsic reward . In order to devise a proper use of this extra freedom , we consider the following two objective functions : η ( π ) = Eτ∼π [ ∑ t γtr ( st , at , st+1 ) ] ( 8 ) η̃ ( π ) = Eτ∼π [ ∑ t γtr ( st , at , st+1 ) ] + c E ( s , a ) ∼π [ DKL ( P ||P ′ ) | ( s , a ) ] ( 9 ) where τ is a sample trajectory , c is a positive constant , and P ( ·|s , a ) and P ′ ( ·|s , a ) are the true transition probability of an environment and its estimation model , respectively . The first objective function η ( π ) is the actual desired expected cumulative return for policy π and the second objective function η̃ ( π ) is the expected cumulative sum of the actual reward and intentionally-added surprise for policy π . We define π∗ and π̃∗ as optimal solutions which maximize the objective functions ( 8 ) and ( 9 ) , respectively . Note that with additional intrinsic reward , the agent learns π̃∗ . Regarding η ( π∗ ) and η ( π̃∗ ) , we have the following proposition : Proposition 1 . Let η ( π ) be the actual expected discounted sum of extrinsic rewards defined in ( 8 ) . Then , the following inequality holds : 0 ≤ η ( π∗ ) − η ( π̃∗ ) ≤ c E ( s , a ) ∼π̃∗ [ DKL ( P ||P ′ ) | ( s , a ) ] ( 10 ) where c is a positive constant . Proposition 1 implies that better estimation of the true transition probability P by model P ′ makes η ( π̃∗ ) closer to η ( π∗ ) , where π̃∗ is learned based on η̃ ( π ) . Thus , for given P we want to minimize E ( s , a ) ∼π̃∗ [ DKL ( P ||P ′ ) | ( s , a ) ] in ( 10 ) over our estimation model P ′ so that we have a tighter gap between η ( π∗ ) and η ( π̃∗ ) , and the policy π̃∗ learned with the aid of surprise intrinsic reward well approximates the true optimal policy π∗ . Regarding this minimization for tight gap , we have the following proposition : Proposition 2 . Let Pφi ( ·|s , a ) , i = 1 , . . . , K be the ensemble of model distributions , and P ( ·|s , a ) be an arbitrary true transition probability distribution . Then , the minimum of average KLD between P ( ·|s , a ) and the mixture model P ′ = ∑ i qiPφi ( ·|s , a ) over the mixture weights { q1 , · · · , qK |qi ≥ 0 , ∑ i qi = 1 } is upper bounded by the minimum of average KLD between P and Pφi over { i } : i.e. , min q1 , ··· , qK E ( s , a ) ∼π̃∗ [ DKL ( P ∣∣∣∣∣∣∣∣ ∑ i qiPφi ) ∣∣∣∣ ( s , a ) ] ≤ min i E ( s , a ) ∼π̃∗ [ DKL ( P ||Pφi ) | ( s , a ) ] . ( 11 ) As seen in the proof of Proposition 2 in Appendix A , mini E ( s , a ) ∼π̃∗ [ DKL ( P ||Pφi ) | ( s , a ) ] provides the tightest upper bound on minq1 , ··· , qK E ( s , a ) ∼π̃∗ [ DKL ( P || ∑ i qiPφi ) | ( s , a ) ] within the class of linear combinations of the individual surprise values { E ( s , a ) ∼π̃∗ [ DKL ( P ||Pφi ) | ( s , a ) ] , i = 1 , 2 , · · · , K } . Propositions 1 and 2 motivate us to use the minimum among the K available individual surprises for our intrinsic reward to reduce the gap between the actual target reward sum η ( π̃∗ ) of the intrinsic reward-aided learned policy π̃∗ and η ( π∗ ) of the true optimal policy π∗ . Note that with the aid of intrinsic reward , we optimize η̃ ( π ) in fact and this makes our policy ( try to ) approach π̃∗ and the sample trajectory approach ( s , a ) ∼ π̃∗ . So , with E ( s , a ) ∼π̃∗ in the right-hand side of ( 11 ) replaced simply by the computable instantaneous sample-based value and DKL ( P ||Pφi ) | ( s , a ) replaced by the approximation ( 5 ) , we propose using the minimum of rjt , int ( st , at , st+1 ) , j = 1 , · · · , K as the single value of intrinsic reward from the K candidates . That is , the agent selects the index j∗ as j∗ = arg min 1≤j≤K rjt , int ( st , at , st+1 ) ( 12 ) where rjt , int is given by ( 7 ) , and the intrinsic reward is determined as rt , int ( st , at , st+1 ) = r j∗ t , int ( st , at , st+1 ) . ( 13 )

Summary: This paper proposes a model-based method for intrinsic rewards based on probabilistic neural network ensembles. For a particular ensemble element, an intrinsic reward is defined as a log-term that measures the deviation in prediction between a Gaussian mixture over all ensemble elements and the particular ensemble element (at the previous update period). The intrinsic reward signal applied in practice is the minimum of the aforementioned quantity w.r.t. all ensemble elements. The authors provide some theoretical motivation for their approach and validate their method in sparse-reward continuous robotics domains (Mujoco), using PPO as reference algorithm. Experiments are averaged over 10 seeds and compared against various other intrinsic reward baselines including PPO without intrinsic motivation. In these experiments, the method provided by the authors seems to dominate over competing approaches.