Adaptive Multi-model Fusion Learning for Sparse-Reward Reinforcement Learning

In this paper , we consider intrinsic reward generation for sparse-reward reinforcement learning based on model prediction errors . In typical model-prediction-errorbased intrinsic reward generation , an agent has a learning model for the underlying environment . Then , intrinsic reward is designed as the error between the model prediction and the actual outcome of the environment , based on the fact that for less-visited or non-visited states , the learned model yields larger prediction errors , promoting exploration helpful for reinforcement learning . This paper generalizes this model-prediction-error-based intrinsic reward generation method to multiple prediction models . We propose a new adaptive fusion method relevant to the multiple-model case , which learns optimal prediction-error fusion across the learning phase to enhance the overall learning performance . Numerical results show that for representative locomotion tasks , the proposed intrinsic reward generation method outperforms most of the previous methods , and the gain is significant in some tasks . 1 INTRODUCTION . Reinforcement learning ( RL ) with sparse reward is an active research area ( Andrychowicz et al. , 2017 ; Tang et al. , 2017 ; de Abril & Kanai , 2018 ; Oh et al. , 2018 ; Kim et al. , 2019 ) . In sparse-reward RL , the environment does not return a non-zero reward for every agent ’ s action but returns a non-zero reward only when certain conditions are met . Such situations are encountered in many action control problems ( Houthooft et al. , 2016 ; Andrychowicz et al. , 2017 ; Oh et al. , 2018 ) . As in conventional RL , exploration is essential at the early stage of learning in sparse-reward RL , whereas the balance between exploration and exploitation is required later . Intrinsically motivated RL has been studied to stimulate better exploration by generating intrinsic reward for each action by the agent itself . Recently , many intrinsically-motivated RL algorithms have been devised especially to deal with the sparsity of reward , e.g. , based on the notion of curiosity ( Houthooft et al. , 2016 ; Pathak et al. , 2017 ) , surprise ( Achiam & Sastry , 2017 ) . In essence , in these intrinsic reward generation methods , the agent has a learning model for the next state or the transition probability of the underlying environment , and intrinsic reward is designed as the error between the model prediction and the actual outcome of the environment , based on the fact that for less-visited or non-visited states , the learned model yields larger prediction errors , promoting exploration helpful for reinforcement learning . These previous methods typically use a single prediction model for the next state or the environment ’ s transition probability . In this paper , we generalize this model-prediction-error-based approach to the case of multiple prediction models and propose a new framework for intrinsic reward generation based on the optimal adaptive fusion of multiple values from multiple models . The use of multiple models increases diversity in modeling error values and the chance to design a better intrinsic reward from these values . The critical task is to learn an optimal fusion rule to maximize the performance across the entire learning phase . In order to devise such an optimal adaptive fusion algorithm , we adopt the α-mean with the scale-free property from the field of information geometry ( Amari , 2016 ) and apply the meta-gradient optimization to search for optimal fusion at each stage of learning . Numerical results show that the proposed multi-model intrinsic reward generation combined with fusion learning significantly outperforms existing intrinsic reward generation methods . 2 RELATED WORK . Intrinsically-motivated RL and exploration methods can be classified mainly into two categories . One is to explicitly generate intrinsic reward and train the agent with the sum of the extrinsic reward and the adequately scaled intrinsic reward . The other is indirect methods that do not explicitly generate intrinsic reward . Our work belongs to the first category , and we conducted experiments using baselines in the first category . However , we also detailed the second category in Appendix H for readers for further work in the intrinsically-motivated RL area . Houthooft et al . ( 2016 ) used the information gain on the prediction model as an additional reward based on the notion of curiosity . Tang et al . ( 2017 ) efficiently applied count-based exploration to high-dimensional state space by mapping the states ’ trained features into a hash table . The concept of surprise was exploited to yield intrinsic rewards ( Achiam & Sastry , 2017 ) . Pathak et al . ( 2017 ) defined an intrinsic reward with the prediction error using a feature state space , and de Abril & Kanai ( 2018 ) enhanced Pathak et al . ( 2017 ) ’ s work with the idea of homeostasis in biology . Zheng et al . ( 2018 ) used a delayed reward environment to propose training the module to generate intrinsic reward apart from training the policy . This delayed reward environment for sparse-reward settings differs from the previous sparse-reward environment based on thresholding ( Houthooft et al. , 2016 ) . ( The agent gets a non-zero reward when the agent achieves a specific physical quantity - such as the distance from the origin - larger than the predefined threshold . ) Pathak et al . ( 2019 ) interpreted the disagreement among the models as the variance of the predicted next states and used the variance as the final differentiable intrinsic reward . Our method is a generalized version of their work as we can apply our proposed fusion method to the multiple squared error values between a predicted next state and all the predicted next states ’ average . Freirich et al . ( 2019 ) proposed generating intrinsic reward by applying a generative model with the Wasserstein-1 distance . With the concept of state-action embedding , Kim et al . ( 2019 ) adopted the Jensen-Shannon divergence ( JSD ) ( Hjelm et al. , 2019 ) to construct a new variational lower bound of the corresponding mutual information , guaranteeing numerical stability . Our work differs from these two works in that we use the adaptive fusion method of multiple intrinsic reward at every timestep . 3 THE PROPOSED METHOD . 3.1 SETUP . We consider a discrete-time continuous-state Markov Decision Process ( MDP ) , denoted as ( S , A , P , r , ρ0 , γ ) , where S and A are the sets of states and actions , respectively , P : S ×A → Π ( S ) is the transition probability function , where Π ( S ) is the space of probability distributions over S , r : S ×A× S → R is the extrinsic reward function , ρ0 is the probability distribution of the initial state , and γ is the discounting factor . A ( stochastic ) policy is represented by π : S → Π ( A ) , where Π ( A ) is the space of probability distributions on A and π ( a|s ) represents the probability of choosing action a ∈ A for given state s ∈ S . In sparse-reward RL , the environment does not return a non-zero reward for every action but returns a non-zero reward only when certain conditions are met by the current state , the action and the next state ( Houthooft et al. , 2016 ; Andrychowicz et al. , 2017 ; Oh et al. , 2018 ) . Our goal is to optimize the policy π to maximize the expected cumulative return η ( π ) by properly generating intrinsic reward in such sparse-reward environments . We assume that the true transition probability distribution P is unknown to the agent . 3.2 INTRINSIC REWARD DESIGN BASED ON MODEL PREDICTION ERRORS . Intrinsically-motivated RL adds a properly designed intrinsic reward at every timestep t to the actual extrinsic reward to yield a non-zero total reward for training even when the extrinsic reward returned by the environment is zero ( Pathak et al. , 2017 ; Tang et al. , 2017 ; de Abril & Kanai , 2018 ) . In the model-prediction-error-based intrinsic reward design , the agent has a prediction model parametrized by φ for the next state st+1 or the transition probability P ( st+1|st , at ) , and the intrinsic reward is designed as the error between the model prediction and the actual outcome of the environment ( Houthooft et al. , 2016 ; Achiam & Sastry , 2017 ; Pathak et al. , 2017 ; Burda et al. , 2019 ; de Abril & Kanai , 2018 ) . Thus , the intrinsic-reward-incorporated problem under this approach is given in most cases as max π { η ( π ) + c E ( s , a ) ∼π [ D ( P ||Pφ ) | ( s , a ) ] } ( 1 ) for some constant c > 0 and some divergence function D ( ·||· ) , where η ( π ) is the cumulative reward associated with policy π , and Pφ is the learning model parameterized by φ that the agent has regarding the true unknown transition probability P of the environment . For the divergence , the mean squared error ( MSE ) between the actual next state and the predicted next state can be used for the error measure when the learning model predicts the next state itself , or alternatively the Kullback-Leibler divergence ( KLD ) between the probability distribution for the next state st+1 and the predicted probability distribution for st+1 can be used when the learning models learn the transition probability . In the case of KLD , the intractable DKL ( P ||Pφ ) | ( s , a ) with unknown P can be approximated based on the 1-step approximation ( Achiam & Sastry , 2017 ) . 3.3 THE PROPOSED ADAPTIVE FUSION LEARNING . We consider using multiple prediction models and the design of prediction-error-based intrinsic reward from the multiple models . Suppose we have a collection of K ( ≥ 2 ) models parametrized by φ1 , · · · , φK to generate K prediction error ( approximation ) values at timestep t as intrinsic reward rjt , int ( st , at , st+1 ) , j = 1 , · · · , K , respectively . The key problem of multi-model prediction-errorbased intrinsic reward design is how to learn φ1 , · · · , φK and how to optimally fuse the K values rjt , int ( st , at , st+1 ) , j = 1 , · · · , K , to generate a single intrinsic reward to be added to the scalar cumulative return for policy update . The considered multi-model fusion structure is shown in Fig . 1 . To fuse the K values for a single reward value , one can use one of the known methods such as average , minimum , or maximum . However , there is no guarantee of optimality for such arbitrary choices , and one fixed fusion rule may not be optimal for the entire learning phase . Let a fusion function be denoted as rint = f ( r 1 int , r 2 int , · · · , rKint ) , ( 2 ) where r1int , r 2 int , · · · , rKint are the K input values and rint is the output value . To devise an optimal adaptive fusion rule , we consider the following requirements for the fusion function f . Condition 1 . The fusion function f varies with some control parameter to adapt to the relative importance of the K input values . We require Condition 1 so that the fusion of the K input values can adapt to the learning situation . When the more aggressive fusion is required at some phase of learning , we want the function f to be more like maximum . On the other hand , when the more conservative fusion is required at other learning phases , we want the function f to be more like minimum . Furthermore , we want this optimal adaptation is learned based on data to yield maximum cumulative return . In addition , we impose the following relevant condition for any reasonable fusion function : Condition 2 . The fusion function f is scale-free , i.e. , f ( cr1int , cr 2 int , · · · , crKint ) = cf ( r1int , r2int , · · · , rKint ) . ( 3 ) Condition 2 implies that when we scale all the input values by the same factor c , the output is the c-scaled version of the fusion output of the not-scaled inputs . Condition 2 is a proper requirement for any reasonable averaging function . The necessity of Condition 2 is explained in detail in Appendix G. Such a fusion function can be found based on the α-mean of positive measures in the field of information geometry ( Amari , 2016 ) . For any K positive1 values x1 , · · · , xK > 0 , the α-mean of x1 , · · · , xK is defined as fα ( x1 , · · · , xK ) = h−1 ( 1 K K∑ i=1 h ( xi ) ) ( 4 ) where h ( x ) is given by the α-embedding transformation : h ( x ) = { x 1−α 2 , if α 6= 1 log x , if α = 1 . ( 5 ) It is proven that the unique class of transformation h satisfying Condition 2 under the twicedifferentiability and the strict monotonicity of h is given by the α-embedding ( 5 ) ( Amari , 2007 ; 2016 ) . Basically , Condition 2 is used to write fα ( cx1 , · · · , cxK ) = h−1 ( 1 K ∑K i=1 h ( cxi ) ) = cfα ( x1 , · · · , xK ) . Taking h ( · ) on both sides yields h ( cfα ( x1 , · · · , xK ) ) = 1K ∑K i=1 h ( cxi ) . Then , taking partial derivative with respect to xi ( 1 ≤ i ≤ K ) on both sides , we can show that the equation ( 5 ) is the unique class of mapping functions ( Amari , 2007 ; 2016 ) . Furthermore , by varying α , the α-mean includes all numeric fusions with the scale-free property such as minimum , maximum , and conventional mean functions ( Amari , 2016 ) . When α = −∞ , fα ( x1 , · · · , xK ) = maxi xi . On the other hand , when α = ∞ , fα ( x1 , · · · , xK ) = mini xi . As α increases from −∞ to∞ , the α-mean output varies monotonically from maximum to minimum . See Appendix B . Hence , we can perform aggressive fusion to conservative fusion by controlling the parameter α . 3.3.1 LEARNING OF α WITH META-GRADIENT OPTIMIZATION In the proposed adaptive fusion , we need to adaptively control α judiciously to maximize the expected cumulative extrinsic return η ( π ) . To learn optimal α maximizing η ( π ) , we use the meta gradient method ( Xu et al. , 2018 ; Zheng et al. , 2018 ) . Optimal α at each stage of learning is learned with the proposed method , and it will be shown that optimal α varies according to the stage of learning . For policy πθ with policy parameter θ , let us define the following quantities . • η ( πθ ) = Eτ∼πθ [ ∞∑ t=0 γtr ( st , at , st+1 ) ] : the expected cumulative sum of extrinsic rewards which we want to maximize . Here , τ is a sample trajectory . • ηtotal ( πθ ) = Eτ∼πθ [ ∞∑ t=0 γt ( r ( st , at , st+1 ) + cfα ( st , at , st+1 ) ) ] : the expected cumulative sum of both extrinsic and intrinsic rewards with which the policy πθ is updated . Here , the dependence of the fusion output fα on ( st , at , st+1 ) through r j t , int ( st , at , st+1 ) is shown with notation simplification . Then , for a given trajectory τ = ( s0 , a0 , s1 , a1 , . . . ) generated by πθ , we update θ towards the direction of maximizing ηtotal ( πθ ) : θ̃ = θ + δθ∇θηtotal ( πθ ) ( 6 ) where δθ is the learning rate for θ . Then , the fusion parameter α is updated to maximize the expected cumulated sum of extrinsic rewards for the updated policy πθ̃ : α̃ = α+ δα∇αη ( πθ̃ ) ( 7 ) 1When an input value to the α-mean is negative due to divergence approximation in some cases , we can use exponentiation at the input stage and its inverse logarithm at the output stage . We used the exponentiation exp ( −x ) at the input stage with input x and the negative logarithm of the α-mean as its inverse at the output stage for actual implementation . In this case , due to the monotone decreasing property of the input mapping : x→ exp ( −x ) , the output is the maximum when α =∞ and is the minimum when α = −∞ . where δα is the learning rate for α . Note that we update the policy parameter θ to maximize ηtotal ( πθ ) so that the updated policy parameter θ̃ is a function of α . Therefore , ∇αη ( πθ̃ ) is not zero and can be computed by chain rule : ∇αη ( πθ̃ ) = ∇θ̃η ( πθ̃ ) ∇αθ̃ ( 8 ) To learn optimal α together with θ , we adopt an alternating optimization method widely used in meta-parameter optimization . That is , we iterate the following two steps in an alternating manner : 1 ) Update the policy parameter θ to maximize ηtotal ( πθ ) . 2 ) Update the fusion parameter α to maximize η ( πθ̃ ) , where θ̃ is the updated policy parameter from Step 1 ) . In this way , we can learn proper α adaptively over timesteps to maximize the performance .

Summary: This paper proposes an intrinsic reward formulation to address the challenge of sparse reward in reinforcement learning. The key idea is to learn multiple models. The prediction error of each model is used as a component of the intrinsic reward. These components are fused using an alpha-mean function. The parameter alpha can be tuned automatically throughout the training process using meta-gradient methods. The method is evaluated on 6 OpenAI continuous control benchmarks (with delayed reward), and demonstrates better performance than several state-of-the-art prior works on intrinsic reward.