Adaptive Multi-model Fusion Learning for Sparse-Reward Reinforcement Learning

In this paper , we consider intrinsic reward generation for sparse-reward reinforcement learning based on model prediction errors . In typical model-prediction-errorbased intrinsic reward generation , an agent has a learning model for the underlying environment . Then , intrinsic reward is designed as the error between the model prediction and the actual outcome of the environment , based on the fact that for less-visited or non-visited states , the learned model yields larger prediction errors , promoting exploration helpful for reinforcement learning . This paper generalizes this model-prediction-error-based intrinsic reward generation method to multiple prediction models . We propose a new adaptive fusion method relevant to the multiple-model case , which learns optimal prediction-error fusion across the learning phase to enhance the overall learning performance . Numerical results show that for representative locomotion tasks , the proposed intrinsic reward generation method outperforms most of the previous methods , and the gain is significant in some tasks . 1 INTRODUCTION . Reinforcement learning ( RL ) with sparse reward is an active research area ( Andrychowicz et al. , 2017 ; Tang et al. , 2017 ; de Abril & Kanai , 2018 ; Oh et al. , 2018 ; Kim et al. , 2019 ) . In sparse-reward RL , the environment does not return a non-zero reward for every agent ’ s action but returns a non-zero reward only when certain conditions are met . Such situations are encountered in many action control problems ( Houthooft et al. , 2016 ; Andrychowicz et al. , 2017 ; Oh et al. , 2018 ) . As in conventional RL , exploration is essential at the early stage of learning in sparse-reward RL , whereas the balance between exploration and exploitation is required later . Intrinsically motivated RL has been studied to stimulate better exploration by generating intrinsic reward for each action by the agent itself . Recently , many intrinsically-motivated RL algorithms have been devised especially to deal with the sparsity of reward , e.g. , based on the notion of curiosity ( Houthooft et al. , 2016 ; Pathak et al. , 2017 ) , surprise ( Achiam & Sastry , 2017 ) . In essence , in these intrinsic reward generation methods , the agent has a learning model for the next state or the transition probability of the underlying environment , and intrinsic reward is designed as the error between the model prediction and the actual outcome of the environment , based on the fact that for less-visited or non-visited states , the learned model yields larger prediction errors , promoting exploration helpful for reinforcement learning . These previous methods typically use a single prediction model for the next state or the environment ’ s transition probability . In this paper , we generalize this model-prediction-error-based approach to the case of multiple prediction models and propose a new framework for intrinsic reward generation based on the optimal adaptive fusion of multiple values from multiple models . The use of multiple models increases diversity in modeling error values and the chance to design a better intrinsic reward from these values . The critical task is to learn an optimal fusion rule to maximize the performance across the entire learning phase . In order to devise such an optimal adaptive fusion algorithm , we adopt the α-mean with the scale-free property from the field of information geometry ( Amari , 2016 ) and apply the meta-gradient optimization to search for optimal fusion at each stage of learning . Numerical results show that the proposed multi-model intrinsic reward generation combined with fusion learning significantly outperforms existing intrinsic reward generation methods . 2 RELATED WORK . Intrinsically-motivated RL and exploration methods can be classified mainly into two categories . One is to explicitly generate intrinsic reward and train the agent with the sum of the extrinsic reward and the adequately scaled intrinsic reward . The other is indirect methods that do not explicitly generate intrinsic reward . Our work belongs to the first category , and we conducted experiments using baselines in the first category . However , we also detailed the second category in Appendix H for readers for further work in the intrinsically-motivated RL area . Houthooft et al . ( 2016 ) used the information gain on the prediction model as an additional reward based on the notion of curiosity . Tang et al . ( 2017 ) efficiently applied count-based exploration to high-dimensional state space by mapping the states ’ trained features into a hash table . The concept of surprise was exploited to yield intrinsic rewards ( Achiam & Sastry , 2017 ) . Pathak et al . ( 2017 ) defined an intrinsic reward with the prediction error using a feature state space , and de Abril & Kanai ( 2018 ) enhanced Pathak et al . ( 2017 ) ’ s work with the idea of homeostasis in biology . Zheng et al . ( 2018 ) used a delayed reward environment to propose training the module to generate intrinsic reward apart from training the policy . This delayed reward environment for sparse-reward settings differs from the previous sparse-reward environment based on thresholding ( Houthooft et al. , 2016 ) . ( The agent gets a non-zero reward when the agent achieves a specific physical quantity - such as the distance from the origin - larger than the predefined threshold . ) Pathak et al . ( 2019 ) interpreted the disagreement among the models as the variance of the predicted next states and used the variance as the final differentiable intrinsic reward . Our method is a generalized version of their work as we can apply our proposed fusion method to the multiple squared error values between a predicted next state and all the predicted next states ’ average . Freirich et al . ( 2019 ) proposed generating intrinsic reward by applying a generative model with the Wasserstein-1 distance . With the concept of state-action embedding , Kim et al . ( 2019 ) adopted the Jensen-Shannon divergence ( JSD ) ( Hjelm et al. , 2019 ) to construct a new variational lower bound of the corresponding mutual information , guaranteeing numerical stability . Our work differs from these two works in that we use the adaptive fusion method of multiple intrinsic reward at every timestep . 3 THE PROPOSED METHOD . 3.1 SETUP . We consider a discrete-time continuous-state Markov Decision Process ( MDP ) , denoted as ( S , A , P , r , ρ0 , γ ) , where S and A are the sets of states and actions , respectively , P : S ×A → Π ( S ) is the transition probability function , where Π ( S ) is the space of probability distributions over S , r : S ×A× S → R is the extrinsic reward function , ρ0 is the probability distribution of the initial state , and γ is the discounting factor . A ( stochastic ) policy is represented by π : S → Π ( A ) , where Π ( A ) is the space of probability distributions on A and π ( a|s ) represents the probability of choosing action a ∈ A for given state s ∈ S . In sparse-reward RL , the environment does not return a non-zero reward for every action but returns a non-zero reward only when certain conditions are met by the current state , the action and the next state ( Houthooft et al. , 2016 ; Andrychowicz et al. , 2017 ; Oh et al. , 2018 ) . Our goal is to optimize the policy π to maximize the expected cumulative return η ( π ) by properly generating intrinsic reward in such sparse-reward environments . We assume that the true transition probability distribution P is unknown to the agent . 3.2 INTRINSIC REWARD DESIGN BASED ON MODEL PREDICTION ERRORS . Intrinsically-motivated RL adds a properly designed intrinsic reward at every timestep t to the actual extrinsic reward to yield a non-zero total reward for training even when the extrinsic reward returned by the environment is zero ( Pathak et al. , 2017 ; Tang et al. , 2017 ; de Abril & Kanai , 2018 ) . In the model-prediction-error-based intrinsic reward design , the agent has a prediction model parametrized by φ for the next state st+1 or the transition probability P ( st+1|st , at ) , and the intrinsic reward is designed as the error between the model prediction and the actual outcome of the environment ( Houthooft et al. , 2016 ; Achiam & Sastry , 2017 ; Pathak et al. , 2017 ; Burda et al. , 2019 ; de Abril & Kanai , 2018 ) . Thus , the intrinsic-reward-incorporated problem under this approach is given in most cases as max π { η ( π ) + c E ( s , a ) ∼π [ D ( P ||Pφ ) | ( s , a ) ] } ( 1 ) for some constant c > 0 and some divergence function D ( ·||· ) , where η ( π ) is the cumulative reward associated with policy π , and Pφ is the learning model parameterized by φ that the agent has regarding the true unknown transition probability P of the environment . For the divergence , the mean squared error ( MSE ) between the actual next state and the predicted next state can be used for the error measure when the learning model predicts the next state itself , or alternatively the Kullback-Leibler divergence ( KLD ) between the probability distribution for the next state st+1 and the predicted probability distribution for st+1 can be used when the learning models learn the transition probability . In the case of KLD , the intractable DKL ( P ||Pφ ) | ( s , a ) with unknown P can be approximated based on the 1-step approximation ( Achiam & Sastry , 2017 ) . 3.3 THE PROPOSED ADAPTIVE FUSION LEARNING . We consider using multiple prediction models and the design of prediction-error-based intrinsic reward from the multiple models . Suppose we have a collection of K ( ≥ 2 ) models parametrized by φ1 , · · · , φK to generate K prediction error ( approximation ) values at timestep t as intrinsic reward rjt , int ( st , at , st+1 ) , j = 1 , · · · , K , respectively . The key problem of multi-model prediction-errorbased intrinsic reward design is how to learn φ1 , · · · , φK and how to optimally fuse the K values rjt , int ( st , at , st+1 ) , j = 1 , · · · , K , to generate a single intrinsic reward to be added to the scalar cumulative return for policy update . The considered multi-model fusion structure is shown in Fig . 1 . To fuse the K values for a single reward value , one can use one of the known methods such as average , minimum , or maximum . However , there is no guarantee of optimality for such arbitrary choices , and one fixed fusion rule may not be optimal for the entire learning phase . Let a fusion function be denoted as rint = f ( r 1 int , r 2 int , · · · , rKint ) , ( 2 ) where r1int , r 2 int , · · · , rKint are the K input values and rint is the output value . To devise an optimal adaptive fusion rule , we consider the following requirements for the fusion function f . Condition 1 . The fusion function f varies with some control parameter to adapt to the relative importance of the K input values . We require Condition 1 so that the fusion of the K input values can adapt to the learning situation . When the more aggressive fusion is required at some phase of learning , we want the function f to be more like maximum . On the other hand , when the more conservative fusion is required at other learning phases , we want the function f to be more like minimum . Furthermore , we want this optimal adaptation is learned based on data to yield maximum cumulative return . In addition , we impose the following relevant condition for any reasonable fusion function : Condition 2 . The fusion function f is scale-free , i.e. , f ( cr1int , cr 2 int , · · · , crKint ) = cf ( r1int , r2int , · · · , rKint ) . ( 3 ) Condition 2 implies that when we scale all the input values by the same factor c , the output is the c-scaled version of the fusion output of the not-scaled inputs . Condition 2 is a proper requirement for any reasonable averaging function . The necessity of Condition 2 is explained in detail in Appendix G. Such a fusion function can be found based on the α-mean of positive measures in the field of information geometry ( Amari , 2016 ) . For any K positive1 values x1 , · · · , xK > 0 , the α-mean of x1 , · · · , xK is defined as fα ( x1 , · · · , xK ) = h−1 ( 1 K K∑ i=1 h ( xi ) ) ( 4 ) where h ( x ) is given by the α-embedding transformation : h ( x ) = { x 1−α 2 , if α 6= 1 log x , if α = 1 . ( 5 ) It is proven that the unique class of transformation h satisfying Condition 2 under the twicedifferentiability and the strict monotonicity of h is given by the α-embedding ( 5 ) ( Amari , 2007 ; 2016 ) . Basically , Condition 2 is used to write fα ( cx1 , · · · , cxK ) = h−1 ( 1 K ∑K i=1 h ( cxi ) ) = cfα ( x1 , · · · , xK ) . Taking h ( · ) on both sides yields h ( cfα ( x1 , · · · , xK ) ) = 1K ∑K i=1 h ( cxi ) . Then , taking partial derivative with respect to xi ( 1 ≤ i ≤ K ) on both sides , we can show that the equation ( 5 ) is the unique class of mapping functions ( Amari , 2007 ; 2016 ) . Furthermore , by varying α , the α-mean includes all numeric fusions with the scale-free property such as minimum , maximum , and conventional mean functions ( Amari , 2016 ) . When α = −∞ , fα ( x1 , · · · , xK ) = maxi xi . On the other hand , when α = ∞ , fα ( x1 , · · · , xK ) = mini xi . As α increases from −∞ to∞ , the α-mean output varies monotonically from maximum to minimum . See Appendix B . Hence , we can perform aggressive fusion to conservative fusion by controlling the parameter α . 3.3.1 LEARNING OF α WITH META-GRADIENT OPTIMIZATION In the proposed adaptive fusion , we need to adaptively control α judiciously to maximize the expected cumulative extrinsic return η ( π ) . To learn optimal α maximizing η ( π ) , we use the meta gradient method ( Xu et al. , 2018 ; Zheng et al. , 2018 ) . Optimal α at each stage of learning is learned with the proposed method , and it will be shown that optimal α varies according to the stage of learning . For policy πθ with policy parameter θ , let us define the following quantities . • η ( πθ ) = Eτ∼πθ [ ∞∑ t=0 γtr ( st , at , st+1 ) ] : the expected cumulative sum of extrinsic rewards which we want to maximize . Here , τ is a sample trajectory . • ηtotal ( πθ ) = Eτ∼πθ [ ∞∑ t=0 γt ( r ( st , at , st+1 ) + cfα ( st , at , st+1 ) ) ] : the expected cumulative sum of both extrinsic and intrinsic rewards with which the policy πθ is updated . Here , the dependence of the fusion output fα on ( st , at , st+1 ) through r j t , int ( st , at , st+1 ) is shown with notation simplification . Then , for a given trajectory τ = ( s0 , a0 , s1 , a1 , . . . ) generated by πθ , we update θ towards the direction of maximizing ηtotal ( πθ ) : θ̃ = θ + δθ∇θηtotal ( πθ ) ( 6 ) where δθ is the learning rate for θ . Then , the fusion parameter α is updated to maximize the expected cumulated sum of extrinsic rewards for the updated policy πθ̃ : α̃ = α+ δα∇αη ( πθ̃ ) ( 7 ) 1When an input value to the α-mean is negative due to divergence approximation in some cases , we can use exponentiation at the input stage and its inverse logarithm at the output stage . We used the exponentiation exp ( −x ) at the input stage with input x and the negative logarithm of the α-mean as its inverse at the output stage for actual implementation . In this case , due to the monotone decreasing property of the input mapping : x→ exp ( −x ) , the output is the maximum when α =∞ and is the minimum when α = −∞ . where δα is the learning rate for α . Note that we update the policy parameter θ to maximize ηtotal ( πθ ) so that the updated policy parameter θ̃ is a function of α . Therefore , ∇αη ( πθ̃ ) is not zero and can be computed by chain rule : ∇αη ( πθ̃ ) = ∇θ̃η ( πθ̃ ) ∇αθ̃ ( 8 ) To learn optimal α together with θ , we adopt an alternating optimization method widely used in meta-parameter optimization . That is , we iterate the following two steps in an alternating manner : 1 ) Update the policy parameter θ to maximize ηtotal ( πθ ) . 2 ) Update the fusion parameter α to maximize η ( πθ̃ ) , where θ̃ is the updated policy parameter from Step 1 ) . In this way , we can learn proper α adaptively over timesteps to maximize the performance .

Summary: In this paper, the authors explore a model based intrinsic reward generation mechanism, in environment settings where the reward assignment is sparse. The authors used an ensemble of models,  and computed the alpha-mean value of their KL divergences with respect to the "true transitions". The alpha-mean serves as the intrinsic reward, with the parameter alpha being co-optimized during the training.  The authors demonstrated that their proposed approach yields top performance in six augmented MuJuCo continuous control tasks.