Adaptive Multi-model Fusion Learning for Sparse-Reward Reinforcement Learning
Authors: Giseung Park
Whiyoung Jung
Sungho Choi
Youngchul Sung
Venue:  (2021)
Keywords: sparse-reward RL
intrinsic reward generation
adaptive fusion
information geometry
scale-free property

Abstract: In this paper, we consider intrinsic reward generation for sparse-reward reinforcement learning based on model prediction errors. In typical model-prediction-error-based intrinsic reward generation, an agent has a learning model for the underlying environment. Then intrinsic reward is designed as the error between the model prediction and the actual outcome of the environment, based on the fact that for less-visited or non-visited states, the learned model yields larger prediction errors, promoting exploration helpful for reinforcement learning. This paper generalizes this model-prediction-error-based intrinsic reward generation method to multiple prediction models. We propose a new adaptive fusion method relevant to the multiple-model case, which learns optimal prediction-error fusion across the learning phase to enhance the overall learning performance.  Numerical results show that for representative locomotion tasks, the proposed intrinsic reward generation method outperforms most of the previous methods, and the gain is significant in some tasks.

Under review as a conference paper at ICLR 2021
ADAPTIVE MULTI-MODEL FUSION LEARNING FOR SPARSE-REWARD REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we consider intrinsic reward generation for sparse-reward reinforce- ment learning based on model prediction errors. In typical nodel-prediction-eroo based intrinsic reward generation, an agent has a learning model for the underlying environment. Then, intrinsic reward designed the error between the model prediction and the actual outcome of the environment, based on the fact that for less-visited or non-visiie non-visited states, the learned model yields larger prediction errors, promoting exploration helpful for reinforcement learning. This paper generalizes this paaadll-rediiiiooerror---eeee
intrinsic reward generation method to multiple prediction models. We propose new adaptive fusion method relevant to the multiple-mode case, which learns optimal prediction-error fusion across the learn- ing phase to enhance the overall learning performance. Numerical results show that for representative locomotion tasks, the proposed intrinsic reward generation method outperforms most of the previous methods, and the gain is significant in some tasks.
INTRODUCTION
Reinforcement learning (RL) with sparse reward is active research area (Andrychowicz al. 2017 Tang et al. 2017 de Abril & Kanai 2018 Oh et al. 2018 Kim et al.] 2019). In sparse- reward RL, the environment does not return non-zero reward for every agent's 's action but returns a non-zere reward only when certain conditions are met. Such situations are encountered in many action control problems (Houthooft et al. 2016 Andrychowicz et al., 2017, Oh al. 2018 As in conventional RI., exploration is essential at the early stage of learning in sparse-reward RI, whereas the balance between exploration and exploitation is required later.
Intrinsically motivated RL has been studied to stimulate better exploration by generating intrinsic reward for each action by the agent itself. Recently, many autrinsically-motivae
RL algorithms have been devised especially to deal with the sparsity of reward, e.g., based the notion of curiosity (Houthooft et al.) 2016 Pathak et al., 2017). surprise (Achiam & Sastry 2017). ). In essence, in these intrinsic reward generation methods, the agent has learning model for the next state or the transition probability of the underlying environment, and intrinsic reward is designed as the error between the model prediction and the actual outcome of the environment, based on the fact that for less-visited or non-visited states, the learned model yields larger prediction errors, promoting exploration helpful for reinforcement learning. These previous methods typically use a single prediction model for the next state or the environment's 's transition probability.
In this paper, we generalize this modee--rediction-ee -erdiittioo-erro-bbseeed approach to the case of multiple prediction models and propose a new framework for intrinsic reward generation based on the optimal adaptive fusion of multiple values from multiple models. The use of multiple models increases diversity in modeling error values and the chance to design a better intrinsic reward from these values. The critical task is to learn an optimal fusion rule to maximize the performance across the entire learning phase. In order to devise such optimal adaptive fusion algorithm, we adopt the a-mean with the scale-free free property from the field information geometry (Amari (016) and apply the meta- gratdgggaiintt optimization to search for optimal fusion at each stage of learning. Numerical results show that the proposed multi-mode intrinsic reward generation combined with fusion learning significantly outperforms existing intrinsic reward generation methods.

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
Intrinsically-motee motivated RL and exploration methods can be classified mainly into two categories. One is to explicitly generate intrinsic reward and train the agent with the sum of the extrinsic reward and the adequately scaled intrinsic reward. The other is indirect methods that do not explicitly generate intrinsic reward. Our work belongs to the first category, and we conducted experiments using baselines in the first category. However, we also detailed the second category in Appendix H for readers for further work in the intrinsically-motiiaeed RL area.
Houthooft et al. (0166) used the information gain on the prediction model as an additional reward based the notion of curiosity. [Tang et al. (0177) efficiently applied count-based exploration to high-dimensional state space by mapping the states' trained features into a hash table. The concept of surprise was exploited to yield intrinsic rewards (Achiam & Sastry (017). Pathak et al.] (2017) defined an intrinsic reward with the prediction error using a feature state space, and de Abril & Kanai (2018) enhanced Pathhkk al. (2017)'s work with the idea of homeostasis in biology.
Zheng et al. (2018) used a delayed reward environment to propose training the module to generate intrinsic reward apart from training the policy. This delayed reward environment for sparse-reward settings differs from the previous sparse-reward environment based on thresholding (Houthooft et al. 2016). (The agent gets a non-zero reward when the agent achieves a specific physical quantity - such as the distance from the origin larger than the predefined threshold.) Pathak et al. (2099) interpreted the disagreement among the models as the variance of the predicted next states and used the variance as the final differentiable intrinsic reward. Our method is a generalized version of their work as we can apply our proposed fusion method to the multiple squared error values between a predicted next state and all the predicted next states' average. Freirich et al. (2019) proposed generating intrinsic reward by applying a generative model with the Wasserstein- distance. With the concept of state-action embedding, Kim et al. (2019) adopted the Gensen-Shannon divergence (JSD) (Hjelm et al. 2019) to construct a new variational lower bound of the corresponding mutual information, guaranteeing numerical stability. Our work differs from these two works in that use the adaptive fusion method of multiple intrinsic reward at every timestep.
3 THE PROPOSED METHOD
3.1 SETUP
We consider a discrete-time continuous-state Markov Decision Process (MDP), denoted as (S, P,r. Po: 7), where S and A are the sets of states and actions, respectively, S:S A + ITS) the transition probability function, where II(S) is the space of probability distributions over S, r: S x A x S > IR is the extrinsic reward function, Pn is the probability distribution of the initial state, and y is the discounting factor. A (stochastic) policy is represented by S IT)A). where I(A) is the space of probability distributions on A and <(aas) represents the probability of choosing action a E AA for given state s E S. In sparse-rewar RL, the environment does not return a non-zero reward for every action but returns non-zero reward only when certain conditions are met by the current state, the action and the next state (Houthooft et al. 2016)] Andrychowicz et al., 2017: Oh et al. 2018). Our goal is optimize the policy ı to maximize the expected cumulative return 77(*²) by properly generating intrinsic reward in such sparse-reward environments. We assume that the true transition probability distribution P is unknown to the agent.
3.2 INTRINSIC REWARD DESIGN BASED ON MODEL PREDICTION ERRORS
intrisically-motiiatedd motivated RL adds properly designed intrinsic reward at every timestep to the actual extrinsic reward to yield a non-zero total reward for training even when the extrinsic reward returned by the environment is zero (Pathak et al. 2017 Tang et al. 2017 de Abril & Kanai 2018). In the paadel-prediciioo-ero---beee
intrinsic reward design, the agent has prediction model parametrized by ó for the next state S141 or the transition probability P( 1 |t, a.), and the intrinsic reward is designed the error between the model prediction and the actual outcome of the environment (Houthooft et al., 2016 Achiam & Sastry 2017; Pathak et al., 2017: ( Burda et al.) 2019 de Abril & Kanai 2018). Thus, the intrinsi-rewand--ncoponaa problem under this approach is given in most
2

---

Under review as a conference paper at ICLR 2021
Pg
(P Pa.)
Inn
Paa
(P Prr)
Tint
Fusion function f
:
:
PaK
D(P Ppx))
Pind
Control
Figure 1: Adaptive fusion of K prediction enrors from the multiple models
cases
max {π((T) + ((ss))=m
(1)
for some constant and some divergence function D(-II-), where r)(T) is the cumulative reward associated with policy and Pa is the learning model parameterized by é that the agent has regarding the true unknown transition probability of the environment. For the divergence, the mean squared error (MSE) between the actual next state and the predicted next state can be used for the error measure when the learning model predicts the next state itself, or alternatively the Kullback-Leible divergence (KLD) between the probability distribution for the next state 5 and the predicted probability distribution for $++1 can be used when the learning models learn the transition probability. In the case of KLD, the intractable D (P) Pg (s. a) with unknown P can be approximated based on the 1 -step approximation (Achiam & Sastry 2017).
3.3 THE PROPOSED ADAPTIVE FUSION LEARNING
We consider using multiple prediction models and the design of prediction-error-bassee intrinsic reward from the multiple models. Suppose we have a collection of K K( 2) models parametrized by b2...., dk to generate K prediction error (approximation) values at timestep as intrinsic reward Ti,se Ge. St+1).j 1,.... K, respectively. The key problem of multi-model prediction-erree based intrinsic reward design is how to learn Φ², .... oit and how to optimally fuse the K values (St, ( $t+1). = 1, 1.... K, to generate a single intrinsic reward to be added to the scalar cumulative return for policy update. The considered multi-model fusion structure is shown in Fig. I To fuse the K values for a single reward value, one can use one of the known methods such as average, minimum, or maximum. However, there is no guarantee of optimality for such arbitrary choices, and one fixed fusion rule may not be optimal for the entire learning phase.
Let fusion function be denoted as
Tint f(r sntt ints
snt)s
where iine r .... , are the input values and rin is the output value. To devise an optimal adaptive fusion rule, consider the following requirements for the fusion function f.
Condition The fusion function f varies with some control parameter to adapt to the relative importance of the K imput values.
We require Condition so that the fusion the K input values can adapt to the learning situation. When the more aggressive fusion is required at some phase of learning, want the function f to be more like maximum. On the other hand, when the more conservative fusion is required at other learning phases, want the function f be more like minimum. Furthermore, we want this optimal adaptation is learned based on data to yield maximum cumulative return. In addition, we impose the following relevant condition for any reasonable fusion function:
Condition 2. The fusion function f scale-free, i.e.,
f(er) 2r ". _erkk ( cf cf 12 iine:" ".. ; m
(3)
3

---

Under review as a conference paper at ICLR 2021
Condition 2 implies that when scale all the input values by the same factor c, the output is the o-scaled version of the fusion output of the not-scaled inputs.
Condition 2 is proper requirement for any reasonable averaging function. The necessity of Condition 2 is explained in detail in Appendix G Such a fusion function can be found based on the a a-mean of positive measures in the field of information geometry (Amari 2016). For any K positive values I1, 1.. IK > 0, the a-mean of Zi," IK is defined as
fa(x1, FK) h-1 (
h(a;)
(4)
where h(z) is given by the a-embedding transformation:
h(x))
i log if
x 2
(5)
is proven that the unique class of transformation h satisfying Condition under the twice- differentiability and the strict monotonicity of h is given by the a-embedding ( (Amari 2007
2016). Basically, Condition 2 is used to write fa(cr1,'.. ,kk = h
K Li= h(cr,)
cf xx)) Taking h(-) both sides yields h(cfa(F1, ... xk)) Ei= M(C;)) Then,
K taking partial derivative with respect to 2; (1 Sis K) on both sides, we can show that the equation (5) is the unique class of mapping functions (Amari 2007 2016).
Furthermore, by varying a, the a-mean includes all numeric fusions with the scale-free property such minimum, maximum, and conventional mean functions (Amari 2016). When
-C, max; 24. On the other hand, when a = xo, fa ... .xk) min; 2;. As a increases from 00 to oc, the a-mean output varies monotonically from maximum to minimum.
fa (.1..... ,"k)
See Appendix B Hence, we can perform aggressive fusion to conservative fusion by controlling the parameter a.
3.3.1 LEARNING OF a WITH META-GRADIES OPTIMIZATION
In the proposed adaptive fusion, we need adaptively control a judiciously maximize the expected cumulative extrinsic return v.) To learn optimal a maximizing 77T)) we use the meta gradient method (Xu et al. 2018 Zhenng al., 2018)) Optimal a at each stage of learning is learned with the proposed method, and it will be shown that optimal a varies according to the stage learning. For policy He with policy parameter let us define the following quantities.
((())) Erawo Σ ('r(st, (1.)))))))))) the expected cumulative sum of extrinsic rewards which we
want to maximize. Here, 7 is a sample trajectory.
(Fo) Erroro Σ (r(s,, 8441) efa ar, 11+1)) the expected cumulative sum of both extrinsic and intrinsic rewards with which the policy we is updated. Here, the dependence of the fusion output fa on (st, de, $141) through (s, ae, $141) is shown with notation simplification.
Then, for given trajectory (So. ao, 81, .1....) generated by ma update 0 towards the direction of maximizing (hotat(TT))
-- 0 + 8y V Vottaaa
(6)
where dg is the learning rate for 0. Then, the fusion parameter is updated to maximize the expected cumulated sum of extrinsic rewards for the updated policy Tg:
ā = 8, (T))
When an input value to the a-mean negative due divergence approximation in some cases, we can use exponentiation at the input stage and its inverse logarithm at the output stage. We used the exponentiation exp(-a at the input stage with input and the negative logarithm of the c-mean as its inverse at the output stage for actual implementation. In this case, due to the monotone decreasing property of the input mapping: exp(-xxp (xx), the output the maximum when a oc and is the minimum when
4

---

Under review as a conference paper at ICLR 2021
where 6, is the learning rate for a. Note that we update the policy parameter 0 to maximize Nooial (To) so that the updated policy parameter i is function of Therefore, VaT)(Ta) is not zero and can be computed by chain rule:
(o)(TT) = V a TT) V.O
(8)
To learn optimal a together with 0, we adopt an alternating optimization method widely used in meta- parameter optimization. That is, we iterate the following two steps in an alternating manner:
1) Update the policy parameter 0 to maximize Mooal (T))-
2) Update the fusion parameter a to maximize n(TT)), where 0 is the updated policy parameter from Step 1).
In this way, we can learn proper adaptively over timesteps to maximize the performance.
3.4 IMPLEMENTATION
We consider the case of D(-II-) D D-1(-1I-) for implementation example (See Appendix for the comparison of KLD and MSE). We use collection of K prediction models Poi.... , Pax. from the j-th model Par. j = -,... K, we have the j-th prediction error, given by
Then,
Ep log P(.|se, 0e) P'((ss, at)
2 Ep log P'(-|se, or) (9)
1.
DKL (
P'(.|st, ac) Pas as)
Pas (-1s,
Note that the j-th model prediction error DKL ) (sr, ar) is lower bounded as (9) for any distribution p', In order to obtain a tight lower bound, p should be learned be close the true transition probability P. For increased degrees of freedom for better learning and estimation, we use the mixture distribution of Pk- Ei= q.P's for p' with the learnable mixing coefficients qi 2 0 and Σx 4 = 1. The mixture model Pk has increased model order for modeling the true beyond single-mode distribution. Then, the prediction error approximation intrinsic reward for the j-th (x
model Pas at timestep is determined r', (8t, ae $t+1) log Paj (1644118,,aa j 1..... K.
Note that each r/,int can be negative although the KLD is always nonnegative.
Although the proposed intrinsic reward generation method can be combined with general RL al- gorithms, we consider the PPO algorithm (Schulman et al.) 2017). popular on-policy algorithm generating a batch of experiences of length L with every current policy. Thus, the exposition below is focused on application to PPO. For the K prediction models Pat.... Pax, we adopt the fully-
factorized Gaussian distribution (Houthooft et al., 2016) Achiam Sastry 2017). Then, Px becomes the class of K K-moaa Gaussian mixture distributions.
We first update the prediction models Ps....., .P. and the corresponding mixing coefficients are independently initialized, and qi 's are set n for all = I.... K. At every batch period of PPO, to jointly learn o² and gs. we apply estimation (MLE) with an L2-norm regularizer with KL constraints (Williams)
.1..... ΩK- In the beginning, the parameters d!.... .'. .o
maximum-likelinee
Rasmussen 2006 Achiam & Sastry 2017):
K Σ PiP (e'1s,
K
maximize #' 15
B( log
Cne
=:Creu
(10)
subject to
B()a))
(Ko Il (Pa Me,as] $ K,
9:
where of the parameter of the i-th model before the update caused by (10), Greg is the regularization coefficient, and x a positive constant. To solve this optimization problem with respect to (Ø"), we apply the method based second-order approximation (Schulman et al./ [015a) For the update of {gi). we apply the EM method proposed in Dempster et al. 1977 and set yi as
gold Po (s'(ss))
q = =E(o,a,')
ΣK 2oh Pas (8')s,)) (1 (1
(11)
5

---

Under review as a conference paper at ICLR 2021
Wa Walker2d
Hopper
InvartedPendullum
3200
2300
1003
2000
2020
2300
1500
Avcrgo
1000
600
520-
Timeslape 94
Timilippt)
HalfChectah
Humanaid
130
Falds
100
Avataga fetum
2.00
-120
-230
-330
200
-500
3.8 100
Pe Ciny
0.2
0.6
0.8 1.0
0.6
TimustopeM)
Timestssi00
Timusioptt)
Figure 2: Performance comparison. All simulations were conducted over ten fixed random seeds. The y-axis in each figure with the title "Average Return" represents the mean value of the extrinsic returns of the most recent 100 episodes averaged over the ten random seeds. Each colored band in every figure represents the interval of - around the mean curve, where is the standard deviation of the ten instances of data from the ten random seeds. In order to give sufficient time steps for each environment, for the three environments in the top row, the experiments were performed for 3M timesteps. For the environments in the bottom row, the experiments were conducted for 1 1M timesteps. (For clarity, the first author of each of the algorithms is shown in the Humanoid plot.)
where oit is the mixing coefficient of the i-th model before the update caused by For numerical stability, we use the Tlg-summ-xx) comn-epp trick for computing (IT) as well Likeeihood defined in (TO) and V Likkellhood In addition, we apply simultancous update of all 6 and q, 's, which was found to perform better than one-by-one alternating update of the K models for the considered case.
The update of policy by using PPO is as follows. Let D be the batch of experiences for training the total policy, i.e., D (st, ae, rtotal S141, T1 S L r total (+2-1), where ae To, (-1s)).
8141 ~ [- st, a.), and plooall is the total reward described below. Here, Te, is the parameterized I 1 (the batch period index 7 is included in To, for clarity). The total reward at timestep for training the policy is given by
policy the batch period 7 corresponding to timestep t..... ,t
rt rootal ac, $141) ro(st, ar sc+1) + Gre,ine(S ag $1+1)
(12)
where ra(s,, ae, 114+) is the actual sparse extrinsic reward at timestep t from the environment, T (St, de; 81+1) is the intrinsic reward at timestep I, and 3 >> is the weighting factor. Here, for actual computation of the intrinsic reward, we further applied two techniques: the 1-step technique and the normalization technique used in Achiamm & Sastry (2017) (which are described in Appendix C.. Then, the policy To, is updated every batch period with D by following the standard PPO procedure based on the total reward (2).. Summarizing the above, we provide the pseudocode of our algorithm, Algorithm which assumes PPO as the base algorithm, in Appendix
RESULTS
4.1 PERFORMANCE COMPARISON
To evaluate the performance, we considered sparse-reward environments for continuous control. The considered tasks were six enviromments of Mujoco (Todorov et al., (0022), OpenAl Gym (Brockman et al. 2016): Walker2d, Hopper, InvertedPendulum, HalfCheetah, Ant, and Humanoid. To implement a sparse-reward setting, we adopted the delay method (Oh et al.) 2018). We first accumulate extrinsic rewards generated from the considered environments for every A timesteps until the episode ends.

---

Under review as a conference paper at ICLR 2021
Then provide the accumulated sum of rewards to the agent the end of the A timesteps the end of the episode, and repeat this process. For our experiments, we set = 40 as used in (Zheng et al. 2018). We compared the proposed method with existing intrinsic reward generation methods by using PPO as the base algorithm. We considered the existing intrinsic reward generation methods: single-model surprise (Achiam & Sastry 2017), curiosity (Pathak et al. 2017). hashing (Tang et al. 2017). and information gain approximation (de Abril & Kanai 20 We also considered the method using intrinsic reward module (Zheng al.) 2018) among the most recent works introduced in Section 2. which uses delayed sparse-reward setup and provides an implementation code. Finally, we compared the proposed fusion with the disagreement method using the variance of multiple predicted next states as the intrinsic reward (Pathak et al. 2019).
For fair comparison, we used PPO with the same neural network architecture and common hyperpa- rameters. We also applied the same normalization technique in Appendix for all the considered intrinsic reward generation methods so that the performance difference results only from the intrinsic reward generation method. In the case of the state-of ofth---rttt-ttt algorithm by Zheng et al. (2018), veri- fied reproducibility for the setup A 40 by obtaining the same result as the reference. (See Appendix for detailed description of the overall hyperparameters for simulations and reproducibility.)
Fig. 2 shows the comparison results. It is observed that the proposed fuseon-based intrinsic reward generation method yields top-level performance. The gain is significant in Hopper and Walker2d, and the performance variance is much smaller than the state-of-the intrinsic reward module method in most cases.
4.2
ABLATION STUDY
HalfCheetah
HalfCheetah
1500
6e2, mn
6w2, propesol
<e2,
Averag Return
1000
Ca2, axg
-20
-50
-500
0.2
0.4 TimestopiM)
146
366
(b)
Walker2d
HalfCheetah
25:00
Km+0
1500
K-0
K-1
2000
1000
Aergagg
1500
Avaraga
1000
500
-500
0.4
0.6
Timestu//M)
(imwstoppM1
(c)
(d)
Figure 3: (a) Learning curve of during the proposed fusion learning in HalfCheetah for 1M timesteps. (b) The performance comparison with static fusion methods. (c. d) Mean performance for 1M timesteps
function of K for (c) Walker2d and (d) HalfCheetah. K 0 means PPO without intrinsic reward, and K 1 means the single-model surprise method. (K' 4 yielded similar performance that of K 3, so we omitted the curve of K for simplicity.)
7

---

Under review as a conference paper at ICLR 2021
4.2.1 LEARNING BEHAVIOR OF FUSION PARAMETER
We investigated how the fusion parameter a changed adaptively during the training. Fig. 3(a) shows the learning curve of the fusion parameter a in HalfCheetah. It is seen that starting from the initial value a 0, the fusion parameter increases until it reaches approximately 5, maintains the level until approximately 180 iterations (0.4 million timesteps), and then decreases monotonically. The proposed fusion learning method takes relatively more aggressive fusion strategies with a being around 5 (but this is not the too aggressive maximum corresponding to oc) in the early stage of learning. Then, the fusion learning takes more and more conservative fusion strategy by decreasing a more and more to large negative values (i.e., towards minimum taking). This observation is consistent with the general behavior of RL that aggressive exploration is essential in the early stage of learning and conservative exploitation has more considerable weight in the later stage of learning.
As seen in Fig. 3(b) in the fixed fusion case, the method using the average has higher performance than that with minimum maximum in the early stage of training. However, the minimum selection method yields better performance than average or maximum at the later stage. It is seen that the proposed adaptive fusion yields the best performance because the proposed adaptive fusion takes advantage of both fast performance improvement in the early stage and high final performance at the end by learning a optimally.
In order to see the difference between the proposed a-fusion learning and other fusion learning method, we considered a fusion method directly using neural networks. the considered method, we designed neural network fusion function fę( .TK) ) of K imputs with (i) linear activation or (ii) nonlinear (tanh) activation. In both cases, f& has a single hidden layer of size 2K. It is observed that our proposed method outperforms the fusion with learned neural networks using the same KLD model error input. See Appendix for the comparison result.
4.2.2 EFFECT OF THE NUMBER OF PREDICTION MODELS
We investigated the impact of the model order K. Since we adopt Gaussian distributions for the prediction models Pat ..... Pax, the mixture Pk is Gaussian mixture for given state-action pair (s, a). According to a recent result (Haarnoj al. 2018)). the model order of a Gaussian mixture need not be too large to capture the true transition probability distribution effectively in practice. Thus, we evaluated the performance for K = 1, 3, 4. Fig. 33c] and 3(d) show the mean performance as a function of K in Walker2d and HalfCheetah. The performance improves as K increases. Once the proper model order is reached, the performance does not improve further due to more difficult model estimation for higher model orders, expected from intuition. From this result, we found that K or seems proper for all the six environments considered in Section 4.1
5 CONCLUSION
In this paper, we proposed a new adaptive fusion method with multiple prediction models for sparse- reward RL. The mixture of multiple prediction models is used to better approximate the unknown transition probability, and the intrinsic reward is generated by adaptive fusion learning with multiple prediction error values. The ablation study shows that the general principle of RL is valid in the adaptive fusion that we need to take a more aggressive strategy in the early stage and less aggressive strategy in the later stage. Numerical results show that the proposed method outperforms existing intrinsic reward generation methods in the considered sparse environments. The proposed adaptive fusion structure useful not only to the specific problem considered here but also to other problems involving numeric fusion with fusion learning.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning, arXiv preprint arXiv:170..01732, 2017.
Shun-ichi Amari. Integration of stochastic models by minimizing a-divergence. Neural computation.
(1110))2780-27966 2007.
Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint
2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL https
7/openrevi ew wwt.ew..weeeeeeeeeee..ee.....e.. www.ww/wwo.weeeee//////// 1d= 1 JJJR55Ym
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754 4765, 2018.
Cédric Colas, Olivier Sigaud, and Pierre- Yves Oudeyer. GEP-PG: Decoupling exploration and exploitation in deep reinforcement learning algorithms. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1039- 1048, Stockholmsmässann Stockholm Sweden, 10-15 Jul 2018. PMLR.
httpttp::/prccceenggsss mir pttpa/////eee////osssssss
Ildefons Magrans de Abril and Ryota Kanai. Curiosity driven reinforcement learning with homeostatic regulation. arXiv preprint arXiv: 1801.07440, 2018.
Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Openai baselines. https://gi com/
openai mattaaibaaeeiieee 2017.
Dror Freirich, Tzahi Shimkin, Ron Meir, and Aviy Tamar. Distributional multivariate policy evaluation and exploration with the bellman gan. International Conference on Machine Learning, pp. 1983-1992. PMLR, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870, Stockholmsmässsan Stockholm Sweden, 10-15 Jul 2018. PMLR. URL httt:://ppett http:://proceetings..... press/ 3.00/haarnojal8b..rr
Godfrey Harold Hardy, John Edensor Littlewood, György Pólya, DE Littlewood, G Polya, et al. Inequalities. Cambridge university press, 1952.
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019. URL https //openreveww.o net/ pnet/forumt
Devon Hjelm, Alex Fedorov, Samuel
Zhang-Wa Wei Hong, Tzu- Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun Yi Lee. Diversity-din driven exploration strategy for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. (0489-10500, 2018.

---

Under review as a conference paper at ICLR 2021
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1 1109-1111, 2016.
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. In International Conference Machine Learning, pp. 3360-3369, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6900, 2014.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. International Conference on Learning Representations, 2018.
URL http:https://ooettt htt.tppss///ppeeeeeeeee net www.:/rrt/ormmmm Id=sJJinnwRR
Anusha Nagabandi, Gregory Kahn, Ronald Fearing, and Sergey Levine. Neural network dy- namics for model-based deep reinforcement learning with model-free fine-tuning. arXiv preprint arXiv: 1708. 02596, 2017.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self- Self-imitation learning, In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 38783888877 38887, Stockholmsmässan Stockholm Sweden, 10-15 Jul 2018. PMLR. URL htttt:::/p/ooceed/dd ngs. ml press/ v880/oh1 8b. html
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), volume 2017, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International Conference on Machine Learning, pp. 5062-5071. 5071, 2019.
Alfréd Rényi et al. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics. The Regents of the University of California, 1961.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: arXiv:/506.000 1506.02438, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (707.06777 2017.
Lior Shani, Yonathan Efroni, and Shie Mannor. Exploration conscious reinforcement learning revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5680-56809 Long Beach, Californi,, USA, 09-15 Jun 2019. PMLR. URLht //proceedings. mlr mrr... press / v /ts//saannl.a. html
Pranav Shyam, Wojciech Jaskowski, and Faustino Gomez. Model-based active exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Interna- tional Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5788, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL 9a.html
http::/proceedin ngs. w.w.rsp......... v97
shyam /
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # Exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2750-2759, 2017.
10

---

Under review as a conference paper at ICLR 2021
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep rein- forcement learning. In AAAI, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026- 5033. IEEE, 2012.
Christopher K. I. Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. MIT Press Cambridge, MA, 2006.
Sirui Xic, Junning Huang, Lanxin Lei, Chunxiao Liu, Zheng Ma, Wei Zhang, and Liang Lin. NADPEX: An on-policy temporally consistent exploration method for deep reinforcement learning. In International Conference on Learning Representations, 2019. URL https:///penrreiee
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning, In Advances in Neural Information Processing Systems, 2396-2407, 2018.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances Neural Information Processing Systems, pp. 4644 2018

---

Under review as a conference paper at ICLR 2021
A ALGORITHM
Algorithm Sparse RL with Fusion Learning with Multiple Prediction Models for Intrinsic Reward
Generation: PPO Case
1: L batch size for policy training, L' batch size for model training.
2: Lmini miiibbtch size for policy training, L minibatch size for model training.
3: N : epoch size for policy training, N' epoch size for model training.
4: MAX the maximum index of batch period I, K the number of prediction models.
5: Initialize the policy T the K transition probability models Pap:-.. Pag. the mixing coeffi-
the fusion parameter a, and the extrinsic value function parameter Ç. 6: Generate trajectories with Te, and store them the initially empty replay buffer M.
cients -..-. :
7: for Batch period = 0,... MAX - do
8: Train Pat.... ... Par by performing gradient updates for TOp, and update Qı. -KK by performing iterations with (II). For each Paj (1 S i S K), we draw batch Dj of size L' randomly and uniformly from M, and perform the updates with minibatches of size mi drawn from D for N' epochs.
9:
for Timestep IL, 1.
ILL L do Collect 8, from the environment and 0₂ with the policy no.-
10:
11:
Collect 8441 and the extrinsic reward ri from the environment and add (St, Qe; 8141) M.
12:
end for
Acquire the intrinsic reward Fr,ine of the current batch D of size L. (Detail is described in Appendix C)
Train ne, by using PPO with the total rewards (12 and minibatch size Liiing for N epochs.
15:
Train a and Ç by using the parameter learning method described in Section 3.3 with the total
rewards 12) and minibatch size T.moni for N epochs.
16: end for
12

---

Under review as a conference paper at ICLR 2021
B AN EXAMPLE OF a-MEAN WITH VARYING a
The a-mean includes all numeric fusions with the scale-free property such minimum, maximum, and conventional mean functions such as arithmetic, geometric, and harmonic mean, by varying a. For example, the o-mean of two values æ1 and 222 as function of is given
max m 22} (maximum) t1 (arithmetic)
if a 0C
if a -1
fa (2)
32 (geometric)
if 1
(13)
(harmonic)
if a = 3
2
min [1, r2 (minimum)
if = 00
The figure below shows the a-mean of œ1 = and Vi = 5 with respect to a. The a-mean can implement all the possible fusion of K values into a single value under the condition of scale-freeness by varying
X2)
X1
X2
x xxss
5.0
4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
-15
-10
-5
10
Figure 4: o-mean of æ1
1 and 22
13

---

Under review as a conference paper at ICLR 2021
C
1-STEP TECHNIQUE AND NORMALIZATION FOR r2 t,int
For actual computation of the intrinsic reward, we first applied the 1-step technique (Achiam & Sastry 2017).
Note that
Px ( at) Pas (51+1)5),
r2 ae. St+1) = log
(14)
where
Pk Σ P:Por.
With the 1-step technique, the batch index for the numerator in (14) is the current batch index ((1) corresponding to time step t, whereas the batch index for the denominator in (14) is the previous batch index (tt) - 1. Hence, with the 1-step technique, r r' (8t, at, S(41) is modified as
Σ Oi Prico (81+1/82,a)) ax)
log
(15)
Poi Poles (88+1 (8,41|88,a:)) az)
Then, this modified j-th model's prediction error value is applied to the a-fusion in Section 3.3
To improve numerical stability, we further applied the normalization technique (Achiam & Sastry 2017) to the output (st, de, 8t-1) of the a-fusion. Thus, the final actual intrinsic reward given current batch D is expressed as
[a(St, at, Se+1) (e salt,a,')) .3 [D]
Pt Ce. S1+1)
(16)
max
where (16) describes the applied normalization.
14

---

Under review as a conference paper at ICLR 2021
D NEURAL NETWORK ARCHITECTURE AND HYPERPARAMETERS
For actual implementation, the code implemented by Dhariwal et al. (011 and Zheng et al. (0188) are used. The policy, the prediction models, the value function of total reward, and the value function of extrinsic reward Vi were designed by fully collecconeected neural networks, all of which had two hidden layers of size (64, 64) (Houthooft et al., 2016 Dhariwal et al.) 2017; Tang et al., 2017). The tanh activation function was used for all of the networks (Achiam & Sastry 2017| Dhariwal et al. 2017). The means of the fully factorized Gaussian prediction models were the outputs of our networks, and the variances were trainable variables that were initialized to (Dhariwal et al. 2017). Other than the variances, all initialization is randomized so that each of the prediction models was set differently
(Kurutach et al.] 2018; Tavakoli et al. 2018). For the implementation of the policy model and the value function, our method and all the considered intrinsic reward generation method used the same code for the module method (Zheng et al. 2018).
Although a recent work (Achiam & Sastry, 2017) used TRPO (Schulman et al. 2015a) as the baseline learning engine, we used PPO (Schulman et al,; 2017), one the eurrently most popular algorithms for continuous action control, as our baseline algorithm. While the same basic hyperparameters as those in the previous work (Achiam & Sastry| (017)) were used, some hyperparameters were tuned for PPO. X for the GAE method (Schulman al. al.][[011_] (0055) was fixed to 0.95, while the discounting factor was set to y = 0.99. The batch size L for the training of the policy was fixed to 2048. For the policy update using PPO. the minibatch size Lmani was set to 64, the epoch number N 10, the value function coefficient 0.5, the clipping constant 0.2, and the entropy coefficient 0.0. The initial values of the learning rates of Adam optimizer (Kingma & Ba 2014) for updating the policy parameter and the extrinsic value function parameter& Ç were fixed to 0.0003 and 0.0001, respectively. The initial value of the learning rate of a was 0.01 for Hopper, HalfCheetah, Humanoid, and InvertedPendulum. and 0.001 for Ant and Walker2d. The three learning rates were linearly decayed as timestep passed so that the values at the end of the training was O. The initial value of a was set to O.
Each of the single-model surprise method, the hashing method, and our proposed method requires a replay buffer. The size of the used replay buffer for all these three methods is I. If the buffer becomes full, the earlier samples are deleted first. Before the beginning of the iterations, 2048 x B samples from real trajectories generated by the initial policy were added to the replay buffer. We set B 40 for our experiments. For the methods not requiring a replay buffer, i... Curiosity, Information Gain, Module, Disagreement, and PPO Only, we ran 2048 x B = 81920 timesteps before measuring performance for fair comparison.
For the prediction model learning, we set the batch size L' = 2048, L'mine = 64, and N' = 4. The optimization (00) was solved based on second-order approximation (Schulman al. 2015a). When K K=1 1, the optimization (TO) reduces to the model learning problem in Achiam & Sastry (2017). In Achiam & Sastry (017)) the constraint constant in the second-order optimization was well-tuned as 0.001. Therefore, we used this value of K not only to the case of K = 1 but also to the case of K 2 2. We further tuned the value of Greg (10) for each environment, and we set Grgg = 0.01 For the information gain method, we need another hyperparameter h which is the weight to balance the original intrinsic reward and the homeostatic regulation term (de Abril & Kanai 2018). We tuned this hyperparameter for each environment and the used value of / is shown in Table For the disagreement method, we used five deterministic models.
Ant
Happer = 0.01
HalfCheetah
Hurnanoid
InvertedPendalum
Walker2d
Cunossty
0.01
3 2= 0.0001
0.1
3 = 0.013
3f 0.03
Hashing
3 0.0001
8 0.000001
8 0.1
0.0001
0.003
Information Gain
3 0.01, A
8 0.1, h
3 0.0001,
A 4
0.01, f
3 0.0001,
3 0.03, A
Single Surprise
3 0.00001
B 0.0003
0.0001
Disagreement
3 0.000003
8 0.003
3 0.003
9 0.0083
9 = 0.001
3 == 0.1
Table 1: Used hyperparameter values.
Table I summarizes the weighting factor B as well as the hyperparameter h in information gain method. The weighting factor 8 in (12) between the extrinsic reward and the intrinsic reward should be determined for all intrinsic reward generation methods. Since each of the considered methods yields different scale of the intrinsic reward, we used the optimized weighting factor / in (12) for each algorithm for each environment by testing 33 according to log scale (Zheng et al. 2018):
15

---

Under review as a conference paper at ICLR 2021
In the single-model surprise method and the proposed method, the proposed method employed the same hyperparameters as the single-moded surprise method. We confirmed that the hyperparameters of the other five methods were well-tuned in the original papers (Pathak et al., 2017 Tang et al. 2017 de Abril & Kanai, 2018 Zheng et al. 2018 Pathak et al. 2019), and we used the hyperparameters provided by these methods.
{1.0,
10- 0.,,0..,,.2,001,,......
0.1,
For the intrinsic reward module method, we checked that the open-source code reproduced results in Zheng et al. (2018), as shown in Fig. 3 'Module 0.01' represents the module method with training using the sum of intrinsic reward and the scaled extrinsic reward with scaling factor 0.01. 'Module O' represents training using intrinsic reward only (no addition of extrinsic reward). Both methods are introduced in Zheng et al. (2018), and we checked reproducibility when B = 0, i.e., we ran 2048 x B timesteps before measuring performance. We observed that our used code yielded the same results as those in /Zhnn et al. (20T8) (InvertedPendullm is not considered in this paper).
Thus, we used this code for the module method with only one change that we ran 2048 timesteps with B = 40 before measuring performance for fair comparison. (Since the range of intrinsic reward from the module method is [-1, 1], intrinsic reward normalization in (16) not needed.) For the module method in performance comparison 4. we selected a better method between 'Module O' and "Module 0.01", assuming B = 40. 'Module O' performed better than 'Module 0.01' in Hopper and Walker2d, and 'Module 0.01 performed better than "Module O' in the other three environments. We also observed that both 'Module O' and 'Module 0.01' performed poorly in InvertedPendulum, which is not considered in Zheng al. (2018). Therefore, we further fine-tuned both of the two scaling coefficients of extrinsic and intrinsic reward and set the best values of 1.0 and 0.003, respectively.
Ant 0)
Hopper (8 - O)
HalfCheetah (8 10=00 0)
Makin.h B Makun Jun!
Averie fottum
-100
2010
2000
-200
1000
503
520
-402
M
-500
0.2
0.0
3.0
0,6
0,6
TimeetepsM)
Timestepe
TtmrentapNi
Humanoid (8 D)
Walker2d (8 0)
PPO ony
200
2300
Meduks. 0
Moduls 0.31
Aurmge Sutum
400
2200
$500
300
Restage
200
100
0.
(imetoppp))
Figure 5: Reproduced mean performance of the module method over 10 random seeds with A 40 when B- = 0.
16

---

Under review as a conference paper at ICLR 2021
E COMPARISON TO DIRECT NEURAL NETWORK-BASED FUSION
Ndlker2d
Hopper
InvertedPendallmm
2520
1002
Avo PeO
2200
2020
003
1500
603
1020
403
200
520
203
0.4
0.5
0.5
1.0
0.2
0.4
0.5
0.6
0.4
0,6
T'medeppi
Timestepi 9
Timetapi
HalfChoctah
Ant
Humanaid
Ascrage Nat.m
1.02
-100
6.0
502
220
-320
200
-103
100
0.2
0.5
0.8
1.0
0.4 0.9
9.8
0.6 Hinustppt))
Timustop(M)
Tinestest0
Figure 6: Performance comparison between our proposed method (blue curve) and fusion with neural network learning with nonlinear activation (green) when K = 2. (Fusion with neural network learning with linear activation performed worse than the nonlinear activation, so we omitted the result.) All simulations were conducted over ten fixed random seeds.
In order to compare our fusion method to the fusion with neural network learning, we designed a of K 2 inputs with (i) linear activation or (ii) nonlinear (tanh) activation. both cases, fe has a single hidden layer of size 2K. Note that our fusion
neural network fusion function fe , .K)
I-a log exp L x)+exp (- 1 =) w
function with a leamable a is explicitly expressed as f(z, y) = 2 for KL.D error approximation inputs by the exponentiation at the input stage. Fig. 6 shows that our method outperforms the fusion with neural network learning using the same KLD model error input.
2
X
(a)
(b)
Figure 7: Visualizations of our fusion function and the fusion method based on neural network learning (both were trained in the Hopper environment). The gray surface is the (.1, x2) input plane, and and the y-axis is the fusion output: (a) the proposed fusion function (blue) and the linear neural network fusion function (green). (b) The proposed fusion function (blue) and the nonlinear neural network fusion function (green).
In Fig. 7 we plot both the proposed fusion function (x(11 and the neural-network- bbased fusion function y = fe (2) after training in the Hopper environment. Figs, 7(a) and 7(b) shows the comparison in the linear and non-linear hyperbolic tangent activation cases, respectively, As expected, in the linear activation case, (æ1, 22, y) forms a hyperplane. Although the hyperplane is fit in best way, still it cannot perform properly as a fusion function. For example, permutation invariance is property of good fusion function. However, a single linear layer network canmot achieve this property although the proposed o-fusion has the permutation invariance property. Now,
17

---

Under review as a conference paper at ICLR 2021
consider the case of non-linear hyperbolic tangent activation. In this case, as seen in Fig. 7(b) the learned fusion function seems bit better than the linear activation case but is not still symmetric around the line 21 22- Hence, it is not permutation invariant either. It seems that the neural- network-based fusion function requires more complexity and more learning time. The proposed adaptive a-fusion structure captures the fusion behavior only by using single parameter a and the corresponding learning is efficient.
18

---

Under review as a conference paper at ICLR 2021
F FUSION WITH DIFFERENT PREDICTION ERRORS: THE MEAN SQUARED ERROR (MSE) CASE
We Iieke
Hopper
InvertedPendullum
2300
Aeamal alodd
Mivual
3003
Avstoge Naum
2010
1300
1320
633
1200
0.0
0.4
0.8
1.0
0.2
0.6
0.6
0.8
1.0
0.2
0.4
6.8
HalfChectah
Art
Humanaid
1500
Depeose Weric
120
Prgenee Wateo
Prppect Matio
1.00
-150
Averge
502
-200
200
-320
200
-503
-430
0.8
0.8
TimestopoM)
Figure 8: Performance comparison between the proposed method using KLD (blue) and the proposed method using MSE (green). All simulations were conducted over ten fixed random seeds.
We compared the proposed method using KLD with the proposed method using MSE. In the MSE method, we trained ten deterministic models predicting the next state. These multiple models differently initialized and independently trained with different batch data as widely done other
works (Nagabandi al., 2017 Kurutach et al. 2018). Then, we calculated the MSE between the actual next state and the predicted next state as the eгror measure.
Fig. shows that the proposed method using KLD outperforms the proposed method using MSE Since the KLD uses joint training among multiple probabilistic models, the KLD method can reflect the underlying dynamics more accurately. On the other hand, independently trained deterministic models were not diverse enough to effectively capture the underlying dynamics as compared to the joint training.
19

---

Under review as a conference paper at ICLR 2021
G NECESSITY OF SCALE-FREE PROPERTY (CONDITION 2)
Note that basically the fusion function is some kind of averaging function. Condition 2 is a proper condition for any reasonable averaging function (Hardy al. 1952). The scale-free property is also called the homogeneous property.
Suppose that the homogeneous property for the function function f is not satisfied and assume the nonhomogeneous relationship:
((Z11,C2)) f 22),
(17)
where d' (+ c) is mapped for each scaling factor for the completeness of the scaling operation.
Now suppose that there exist two constants c, e' such that e > and 0 < d' < and f(e cx2) = ( 22) for two input 21 and Ø2- Then, the monotonicity is broken. The two inputs 21 and 22 increased as CTj and er2 but the corresponding output is reduced as c' f( as compared to the the situation that we want. Furthermore, by repeatedly applying
original output (x11, This
equation 17 we have
lim [f(c 21, x1, = (c')" (cd")(((())))) f(11, 1-+000
(18)
yielding
000 = 0. f(I11, x2) = 0.
(19)
So, we have contradiction. In the case of < << 1 and > 1, we have a similar contradiction:
f(0,0) = 0o - f(21, 02) = co.
(20)
Now assume that there exist e, d' > and + d', Set two inputs as I2 = x. Then, we have
(21)
((cx) = [cx, cr) = cff(( = d'g(z)
(22)
By repeating the iteration, we have
g(c"x)= = g(x)
(23)
In equation 13 the input to the function g(x) is exponentially increasing as c and the output increases exponentially (c)" Such function g(x) is uniquely given by the form
g(x)
(24)
where means the scaling equivalence. However, for different pair and 8, we also require
g(x) ~ (2))
(25)
The two functions in 24 and 25
cannot be the same in general. Furthermore, g(x) should be the
same for all pairs (c, c This cannot be satisfied in general. So, we have an indiscrepancy in the
nonhomogeneous case. similar situation happens for c, d' 1. However, note that if we have = c' for all scaling factor c, then the two functions in have
24 and 25 are consistent. In this case, we
g(x) (c) log.
(26)
and this makes sense because if the input values are all the same, the output should be the same as the input. Please note that the generalized mean or q-mean exactly satisfies the scaling behavior (26
20

---

Under review as a conference paper at ICLR 2021
H EXPLORATION WITHOUT INTRINSIC REWARD GENERATION
Recent indirect methods can further be classified mainly into two groups: (i) revising the original ob- jective function to stimulate exploration, exploiting intrinsic motivation implicitly, and (ii) perturbing the parameter space of policy.
In the first group, Andrychowicz et al. (017) suggested additional sampling states from the replay buffer and setting those data as new goals for sparse and binary extrinsic reward environments, In their work, the policy was based on the input both state and goal. In our work, on the other hand, the goal concept is not necessary. Oh et al. 20188) proposed that exploration can be stimulated by exploiting novel state-action pairs from the past and used sparse-reward environments by delaying extrinsic rewards. [Hong et al. (2018) revised the original objective function for training by considering the maximization of the divergence between the current policy and recent policies, with an adaptive scaling technique. The dropout concept was applied to the PPO algorithm encourage the stochastic behavior of the agent episode-wisely (Xie et al. 2019). Convex combination of the target policy and any given policy is considered new exploratory policy (Shani al. 2019), which corresponds to solving a surrogate Markov Decision Process, generalizing usual exploration methods such as e-greedy Gaussian noise.
In the second group. Colas et al. 2018) proposed goal-based exploration method for continuous control, which alternates generating parameter-outcome pair and perturbing specific parameters based on randomly drawn goal from the outcome space. Recently, inspired by Chua et al. (2018), (Shya al. (2099) considered pure exploration MDP without any extrinsic reward with the notion utility, where utility is based on JSD and the censen-Rény: divergence (Rényi et al. 1961). They considered several models for the transition functions this work, but they used this compute utility based on the multiple models' average entropy.
21


Reviews:
[{"review_id": "yTyzf5Fs9a4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an intrinsic reward formulation to address the challenge of sparse reward in reinforcement learning. The key idea is to learn multiple models. The prediction error of each model is used as a component of the intrinsic reward. These components are fused using an alpha-mean function. The parameter alpha can be tuned automatically throughout the training process using meta-gradient methods. The method is evaluated on 6 OpenAI continuous control benchmarks (with delayed reward), and demonstrates better performance than several state-of-the-art prior works on intrinsic reward.\n\nOverall, the paper is tackling an important challenge of reinforcement learning. Thus its potential impact is large. The paper is also well written and the results are promising. My main concern is that one important baseline is missing: [Pathak 2019]: \"Self-supervised exploration via disagreement.\" This prior work is very relevant. It also used multiple models and the implementation was so much simpler than the proposed method in this paper. In addition, this paper also claims to be a generalization of [Pathak 2019]. Thus, adding [Pathak 2019] into the baselines for comparisons seems necessary.\n\nI am also curious why not use MSE to measure the model-prediction error? The KL Divergence used in this paper introduces a lot of complexities, both in derivation and implementation (e.g. eq. 9, 10, 11). If MSE is not compatible with the proposed method, more discussions in the paper is welcome. Otherwise, adding an ablation to compare MSE with KLD would help, especially if KLD yields better results than MSE. This would justify the extra complexity introduced by using KLD.\n\nThe need of Condition 2 in Section 3.3 is not sufficiently motivated. How does this condition affect derivation, implementation and final results? It would be great to have a few sentences to clarity this.\n\nIn summary, this is a good paper. However, missing an important baseline [Pathak 2019] seems a significant oversight. I hope that this baseline will be added in the future version of this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "R4rxPzX52-V", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe papers looks at the problem of using intrinsic rewards to help agent explore in sparse reward settings. The paper proposes combining multiple intrinsic rewards and proposes a meta-gradient based method to learning the fusion of these intrinsic rewards.\n\n\nStrengths\n1. Learning in a sparse rewards setting is a challenging and relevant problem to the community.\n2. The paper is well written \n3. The idea of learning to use multiple intrinsic rewards and using different combinations of them at different times seems to be helpful in the experiments\n\nWeaknesses/Comments/Questions\n1. My main concern is that the contribution of the paper might not be sufficient. The main contribution claimed by the paper to me seems like the idea of combining different intrinsic rewards and also learning the hyper-parameter that controls the fusion. In practice different intrinsic rewards are indeed used together and varied at different times. Also learning RL related hyperparameters using meta-gradient (Meta-gradient RL) is not new.\n2. I don't think Zheng et al is really meant for sparse reward settings. So not sure if that's the right SOTA baseline to compare with. Zheng et al is not expected to learn intrinsic rewards that guide initial exploration as it needs extrinsic feedback to learn these intrinsic rewards, else they are just random.\n3. It would be good if the paper evaluates on some domains that are naturally sparse reward setting instead of converting a dense reward setting to a sparse reward setting by accumulating and delaying rewards.\n4. If the main contribution of the paper is on how to fuse, then simple methods of fusing like learning a lenear combination or just a single layer neural network over the different model prediction errors to output the intrinsic reward to use all seem reasonable. Comparison with those would be good.\n5. The conditions that lead to the particular choice of fusion should be verified using ablation study to see if they indeed lead to the performance boost expected or if the performance comes from something else.\n6. Clarification: Is it that all the experiments only fuse between two intrinsic rewards (prediction errors)? If the paper is about the importance of combining different prediction errors, it would be nice to have more than 2 and also some interestingly different model predictions and also analysis and comparison on what happens if we just use them separately. \n\nQuestions to authors.\n1. Please respond to the comments above. Thanks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7Wjoxp1yOfU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors explore a model based intrinsic reward generation mechanism, in environment settings where the reward assignment is sparse. The authors used an ensemble of models,  and computed the alpha-mean value of their KL divergences with respect to the \"true transitions\". The alpha-mean serves as the intrinsic reward, with the parameter alpha being co-optimized during the training.  The authors demonstrated that their proposed approach yields top performance in six augmented MuJuCo continuous control tasks. \n\n\nPros:\n\n(1) A clearly written paper and easy to read and understand. \n\n(2) The approach demonstrated consistent top performance in all benchmarks.  \n\nCons:\n\n(1) Although the author provides a few ablation studies. I propose to add at least one more study that compares the effectiveness of their method under different \"sparsity\" settings. In all experiments, the authors used a fixed interval of 40, at which extrinsic rewards are computed. I am interested to see if their conclusion can still hold when this interval is set at, for example, 10, 20, 60, etc.\n\n(2) The alpha-mean method is one type of scale invariant transformation. I wonder if the authors have studied other transformation functions, or even scale invariant kernel functions?\n\n(3) There are other sparse-reward tasks such as pushing/sliding/pick-and-place, which are under a different category than MuJuCo continuous control tasks. I am interested in seeing if the proposed approach can still perform in these settings. This will also help make the author's statements more convincing. \n\nConclusion:\nPlease address my concerns raised in the \"cons\" section. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5DxdHbKnlHD", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a generalization of the model-prediction-error-based intrinsic reward method by fusing predictions from multiple models.\nThe authors considered the sparse reward scenario in reinforcement learning.\n\nIn related work, the authors mentioned that previous works on image spaces are not directly related to theirs. However, I did not understand what are the limitations or the caveats of the proposed method that leads to this conclusion.\n\nAlso in related work, I did not understand the purpose of detailing not related work in Sec. 2.2., I believe that the authors could use this space to discuss, for instance,\napplications of fusion methods in reinforcement learning scenarios, such as: “Data fusion using Bayesian theory and reinforcement learning method”.\nAnother option, even better, is discussing approaches proposed in the line of investigation using ensembles\n(e.g. “Model Ensemble-Based Intrinsic Reward for Sparse Reward Reinforcement Learning”).\n\nI would like to understand the computational costs involved in using such a fusion approach., both in terms of individual methods and alpha optimization.\n\nIn Fig. 2, it is hard to conclude that the gain over Module baseline is significant only considering the figure. I don’t know if the authors are considering some\nstatistical test when they mention significance, if this is the case, they should properly present the test and premisses.\n\nIn Sec.4.2.4, it is hard to see how the performance improves as K increases. First, only four values for K are a limitation to conclude this. \nAdditionally, for instance, when considering the Walker2d dataset, the performance for K=2 is better than K=3.\n\nReferences should point to the published work rather than the arxiv entries, when the former is available (e..g. Exploration by random network distillation, ICLR’19).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends previous work on intrinsic reward design based on curiosity or surprise toward multiple intrinsic rewards based multiple model predictions and fuse the reward using meta-gradient optimization.  While most reviewers find the paper clearly written, several reviewers do bring up the concern on limited contribution of the work on top of existing ones. Reviewers also would like to see experiments conducted in environment with sparse reward rather than the delayed reward setting constructed from dense reward environments. More ablation studies on the different design choices will also be helpful. 
Decision: Reject