Model Ensemble-Based Intrinsic Reward for Sparse Reward Reinforcement Learning
Authors: Giseung Park
Whiyoung Jung
Sungho Choi
Youngchul Sung
Venue:  (2020)
Keywords: Reinforcement Learning
Intrinsic Reward
Dynamics Model
Ensemble

Abstract: In this paper, a new intrinsic reward generation method for sparse-reward reinforcement learning is proposed based on an ensemble of dynamics models. In the proposed method, the mixture of multiple dynamics models is used to approximate the true unknown transition probability, and the intrinsic reward is designed as the minimum of the surprise seen from each dynamics model to the mixture of the dynamics models. In order to show the effectiveness of the proposed intrinsic reward generation method, a working algorithm is constructed by combining the proposed intrinsic reward generation method with the proximal policy optimization (PPO) algorithm. Numerical results show that for representative locomotion tasks, the proposed model-ensemble-based intrinsic reward generation method outperforms the previous methods based on a single dynamics model.

Under review as a conference paper at ICLR 2020
MODEL ENSEMBLE-BASER INTRINSIC REWARD FOR SPARSE REWARD REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
In this paper, new intrinsic reward generation method for sparse-reward rein- forcement learning is proposed based on an ensemble of dynamics models. In the proposed method, the mixture of multiple dynamics models is used to approximate the true unknown transition probability, and the intrinsic reward is designed as the minimum of the surprise seen from each dynamics model the mixture of the dy- namics models. In order to show the effectiveness of the proposed intrinsic reward generation method, a working algorithm is constructed by combining the proposed intrinsic reward generation method with the proximal policy optimization (PPO) algorithm. Numerical results show that for representative locomotion tasks, the proposed moodel-ensemble-ee the previous methods based on a single dynamics model.
intrinsic reward generation method outperforms
INTRODUCTION
Reinforcement learning (RL) with sparse reward is an active research area (Andrychowicz et al. 2017 de Abril & Kanai 2018 Kim al., [0118) Oh al. 2018 Tang et al. 2017). In typical model-free RL, an agent learns policy to maximize the expected cumulative reward under the circumstance that the agent receives non-zero reward from the environment for each action of the agent. On the contrary, in sparse reward RI., the environment does not return a non-zero reward for every action of the agent but returns a non-zero reward only when certain conditions are met. Such situations are encountered in many action control problems (Andrychowicz et al. 2017 Houthooft et al.) 2016 Oh et al., 2018). As in conventional RL, exploration is important at the early stage of learning in sparse reward RL, whereas the balance between exploration and exploitation is required on the later stage. Methods such as the c-greedy strategy (Mnih et al. 2015 Van Hasselt et al., 2016) and the control of policy gradient with Gaussian random noise (Duan al.]| 2016 Scchulman et al. (015a) have been applied to various tasks for exploration. However, these methods have been revealed to be insufficient for successful learning when reward is sparse (Achiam & Sastry 2017).
In order to overcome such difficulty, intrinsically motivated RL has been studied to stimulate better exploration by generating intrinsic reward for each action by the agent itself, even when reward is sparse. Recently, many intrinsically-motivae the sparsity of reward, e.g., based on the notion of curiosity (Houthooft al. 2016 Pathak al. 2017) and surprise (Achiam & Sastry 2017). It is shown that these algorithms are successful and outperform the previous approaches. In essence, these algorithms use single estimation model for the next state or the environment dynamics to generate intrinsic reward.
RL algorithms have been devised to deal with
In this paper, order to further improve the performance of sparse reward model-free RL,, we propose new method to generate intrinsic reward based on an ensemble of estimation models for the environment dynamics. The rationale behind our approach is that by using a mixture of several distributions, we can increase degrees of freedom for modeling the unknown underlying model dynamics and designing a better reward from the ensemble of estimation models. Numerical results show that the proposed model-ensemble based intrinsic reward generation method yields improved performance as compared to existing reward generation methods for continuous control with sparse reward setting.

---

Under review as a conference paper at ICLR 2020
2 THE PROPOSED METHOD
2.1 PRELIMINARIES
In this paper, we consider a discrete-time continuous-state Markov Decision Process (MDP), denoted as (S, A. r, Po: 7). where S and A are the sets of states and actions, respectively, P S Ax [0, 1] is the transition probability function (called model dynamics), r : SxAxS + R is the extrinsic reward function, Po S [0. 1] is the distribution of the initial state, and ry is the discounting factor. (stochastic) policy is represented by x S x A - [0, 1], where π (a ()) represents the probability of 8. In sparse reward RI., the environment does not return
choosing action A for given state a non-zero reward for every action but returns a non-zero reward only when certain conditions are met by the current state, the action and the next state (Andrychowicz et al. 2017; Houthooft et al. 2016, Oh ct al. 2018). The goal of this paper is to optimize the policy π to maximize the expected cumulative return () (T) by properly generating intrinsic reward such sparse reward environments. We assume that the true transition model P is unknown to the agent.
caatnsically-moivvattt RL adds properly designed intrinsic reward the actual extrinsic reward to yield non-zero total reward for training even when the extrinsic reward returned by the environment zero (de Abril & Kanai 2018 Pathak et al.) 2017: Tang al., (017). One way to design such an intrinsic reward for action control is based surprise, which is a measure of the unexpectedness of observing the next state for a given current state and action pair and is especially useful to yield better exploration (Achiam & Sastry 2017). In this context, recent work (Achiam & Sastry 2017) proposed a promising direction of intrinsic reward design in which the agent tries to optimize its policy ı according to
max {T(()) (s) (Pkssaa)((KK(PPPPPP [PKz(
for some constant e > O, where P& is the learning model parameterized by á that the agent has regarding the true unknown transition probability P of the environment. Dxr (Pl)Pa) the Kullback-Leibler divergence (KLD) between two distributions P and Ps of the next state for given current state-action pair (s, a). and (eaa)< is the expectation over (s, a) following the policy Thus, the surprise is quantified as B(e.o)=x (Dxx (P]) a)] (Achiam & Sastry 2017). Furthermore, the KLD D Dxr(P] Pp) ar) at timestep can be lower-bounded as
is
DKL DKL 0z) E log Par(-1s,, at) Po(+(s,,a))
with an arbitrary choice of the parameter B. Therefore, the intrinsic reward timestep determined (
as T' (St, ae, $t+1) log Polec+1 a..),) Achiam & Sastry 2017), where P&r needs to be designed properly. With d' = ((t) and & = ). where Potty and Po(t-) are respectively the agent's 's model for P timestep and the model before the update at timestep the intrinsic reward is given by the computable quantity named as the 1-step surprise:
1 1-step (e a2 (141)) = log Pacn (1441155)) Eaant
(8c, Poct- 18441184 ax)
The proposed 1-step intrinsic reward performs well compared the previously designed intrinsic reward, and it is based on a single model Ps for P, where Po for given (s, a) is modeled as Gaussian distribution (Achiam & Sastry 2017).
2.2 INTRINSIC REWARD DESIGN FROM AN ENSEMBLE OF DYNAMICS MODELS
In this paper, we take the principle that DKz (s) a) a reasonable measure for surprise to promote exploration, and generalize the intrinsic reward design under this measure. However, instead of using single learning model for P as the previous approach, we propose using an ensemble of K dynamics models Pot,-.. Pox for P. constructing the mixture distribution
K
PK
Σ
q:Pa
(4)
i=1
2

---

Under review as a conference paper at ICLR 2020
with the mixing coefficients 9: 2 0 and Σ 9: 1, and using Pk in (4) as estimate for the true unknown P. The rationale behind this is that by using a mixture of several distributions we increase degrees of freedom for modeling the underlying model dynamics and designing better intrinsic reward. For the j-th model Pas.j = =,,.. K, we have
Pk (x(()st,ar)
DKL (PI(Pl (((s, ar) 2 E log
(5)
Poo (1) ar)
as in Thus, for Pas, the intrinsic reward at timestep determined as
Pk Px (8 Por (s (s,
d tiint (8t, 0o: 8141) log
(6)
Furthermore, (6) can be modified to yield 1-step surprise intrinsic reward as
E Pifee (st+1/st,
Pl.ine
(St, (St,
az 5c+1) log
Pol P (514 18,, ae)
where Pales and Pol shon- are the j-th model the update period corresponding to timestep and the previous update period I respectively (I(t) will become clear in the subsection 2.3
Since the mixture model 4 has the increased model order for modeling the underlying dynamics distribution beyond single-mode distributions, we have more freedom to design intrinsic reward. That is, we now have values, ri, (8,, at, 8++1). j = -,.... K, for candidates for intrinsic reward. In order to devise a proper use of this extra freedom, we consider the following two objective functions:
(TT)) Eynx
x' 2r,(s,,0,, s141)
(8)
(TT) Er
+ c (ssa)) [DKl (ss
where is sample trajectory, transition probability of an environment and its estimation model, respectively. The first objective function ()TT) is the actual desired expected cumulative return for policy т and the second objective function 7)(T) is the expected cumulative sum of the actual reward and intentionally-adder surprise for policy π. We define π² and #* as optimal solutions which maximize the objective functions (8)
positive constant, and P(+ss, a) and P'(-|s, are the true
and (9) respectively. Note that with additional intrinsic reward, the agent learns A Regarding 17(
and r7 we have the following proposition:
Proposition 1. Let nT(x) be the actual expected discounted sum of extrinsic rewards defined in { Then, the following inequality holds:
0 I n((=)) n(#)) S E(s,a)= [DKL
(10)
where c is a positive constant.
Proposition 1 Iimplies that better estimation of the true transition probability P by model P' makes 7(*") closer to (TT"), where 7 is learned based on 7(T)). Thus, for given P we want to minimize B(s.a)=. DKL (PI PP') [(s, a)] in (10) over our estimation model p so that we have a tighter gap between ()(") ) and r)(*), and the policy 5* learned with the aid of surprise intrinsic reward well approximates the true optimal policy πτ. Regarding this minimization for tight gap, we have the following proposition:
Let (o i = .....,KK be the ensemble of model distributions, and P(-|s, a) be an arbitrary true transition probability distribution. Then, the minimum of average KLD between P(-Is, a) and the mixture model P' Σ, q. (-Is, a) over the mixture weights {91, , 9K (4 2 0, 2. = 1} is upper bounded by the minimum of average KLD between P and P... over {i}: ie.
Proposition 2.
min E(s,a)=*
DKL
P
Σ q.P
(5,0)
I min E(saaa [DKL (P) [(, a)
01 9K
(11)
3

---

Under review as a conference paper at ICLR 2020
As seen in the proof of Proposition 2 in Appendix A min, B ((aa) [xL (PIIPs)) a)) provides the tightest upper bound on ming..--w .n (((sa)>=ā- (DEL(P]) || E. (s,,, a)] within the class of linear combinations of the individual surprise values {)(+,))= [DxL (PIIP^) [(s, i 1,2..... Kj. Propositions I and motivate us to use the minimum among the K available individual surprises for our intrimsic reward to reduce the gap between the actual target reward sum 7(#*) of the intrinsic reward-aides learned policy #* and nnTT") of the true optimal policy x². Note that with the aid of intrinsic reward, we optimize v)())) in fact and this makes our policy (try to) approach #* and the sample trajectory approach (s,a) n²". So, with ((s,a)= in the right-hand side of (11 replaced simply by the computable instantaneous sample-basee value and DEL (PIIP&) [(s, a) replaced by the approximation ( we propose using the minimum of rl,,ne (St, at, 1,+1), = 1, KK as the single value of intrinsic reward from the K candidates. That is, the agent selects the index j* as
j = arg min ri (,,ine at 8141)
(12)
I5iSK
where Pi is given by 7 and the intrinsic reward is determined
(St, Ots 81+1) = ri, (8t, at 8t-1).
(13)
2.3 IMPLEMENTATION
For the dynamics models Pat.... PAx. we adopted the fully-factorized Gaussian distributions (Achiam & Sastry 2017 Houthooft et al. 2016). Then, Px in 44) becomes the class of K-modal Gaussian mixture distributions.
We first update the model ensemble Pat,... P&x and the corresponding mixing coefficients 21-- IK- At the beginning, the parameters d² ..... RK are independently initialized, and q 's set to K for all = ..... K. At every batch period I, in order to jointly learn ø² and qt, we apply maximum-likelihood estimation with an L2-norm regularizer with KL constraints (Achiam & Sastry 2017 Williams & Rasmussen 2006]:
1
K Σ
maximize @ ISiSK
E log
Σ ye Pa (b'l0,))
110-1/2
Likeelhood + a &regalrarierr
(14)
K Sx (1 (isis K) and Σ
subject to
E(saa) (Ki || (s.
9 =1
where da is the parameter of the i-th model before the update 14 a is the regularization coefficient, and λ is a positive constant. To solve this optimization problem with respect to {o"), we apply the method based on second-order approximation (Schulman et al. 2201 For the update of {g.}. we apply the method proposed by Dempster et al. (1977) and set g; as follows:
ge Por (s'ls,
li = (s,a.a))) Σラ=1 0g Pas (8
(1 (ISisK)
(15)
where g.dd the mixing coefficient of the i-th model before the update 15
For numerical stability, we use the "log-sum-expp" trick for computing (15) well as Likelihoodd and V di Likelhhood In addition, we apply simultaneous update of all o and q, s, which was found to perform better than one-by-one alternating update of the K models for our problem.
Although the proposed intrinsic reward generation method can be combined with general RL algorithms, we here consider the PPO algorithm (Schulman et al.] 2017). which is popular on-policy algorithm and generates
batch of experiences of length L with ev- ery current policy. Let D be the batch of experiences for training the policy, i.e., D = Te, (-16)), 8e+1
(st, at, rt,rtota "841, ................................ (+11-2 814 (4+41-111 1, r(+L-)) where at P(-|st, at), and poooall is the total reward (16). Here, no. is the parameterized policy at the batch period I corresponding to timestep ... - 1 (the batch period index I is now included in no for clarity). The total reward at timestep for training the policy is given by
r lotal (St, 0t, 8141) re(S,, at 8141) B (St, (8t, at 8(+1)
(16)
4

---

Under review as a conference paper at ICLR 2020
Algorithm Sparse RL with Model Ensemble- Intrinsic Reward Based on PPO
1: L : batch size for policy training, L' batch size for model training.
2: Imini minibatch size for policy training, miii minibatch size for model training.
3: N epoch size for policy training, N' epoch size for model training.
4: MAX: : the maximum index of batch period I, K : the number of dynamics models.
5: Initialize the policy Th. the K transition probability models po...... Pok. and the correspond-
ing mixing coefficients ç1,-.. ,9K-
6: Generate trajectories with Te, and add them to the initially empty replay buffer M.
7: for Batch period
0,-.. O...... .AX do
8: Train Pat pt..... Pare by performing gradient updates for 14). and update Q1.". . by performing iterations with (15). For this, draw batch D' of size L' randomly and uniformly from M, and perform the updates with minibatches of size m'ms drawn from D for N" epochs.
9:
for Timestep t IL, +, + do
Collect 8, from the environment and a, with the policy me.
11:
Collect 11+1 and the extrinsic reward ry from the environment and add (st, 0z. 81+1) to M.
end for
13:
Calculate the preliminary intrinsic reward Tt,int in 13)
14:
Acquire the normalized intrinsic rewards of the current batch D size by using 17
15:
Train no, by using PPO with the total rewards (T)) and minibatch size Limini for N epochs.
16:
end for
where re( at St+1) is the actual sparse extrinsic reward at timestep t from the environment, inn St, Oe, 9t4-1) is the normalized intrinsic reward timestep i, and / >> 0 is the weighting factor. Note that the actually-used intrinsic reward in, (8r; Ot, 8141) is obtained by applying normalization (Achiam & Sastry 2017) to improve numerical stability
Tt,int (8t, 8(+1)
Fi,ina (St, Oz, 8141)
(17)
E Dean Te (8,0,5)
max
D
where the unnormalized intrinsic reward Tt, T', (S,, ag S1+1) is given by 11 Then, the policy Te, can be updated at every batch period with D by following the standard PPO procedure based on the total reward (16). Summarizing the above, we provide the pseudocode of our algorithm in Algorithm| [ which assumes PPO as the background algorithm. Note that the proposed intrinsic reward generation method can also be applied to other RL algorithms.
3 RESULTS
3.1 EXPERIMENTAL SETUP
In order to evaluate the performance, we considered sparse reward environments for continuous control. The considered tasks were five environments of Mujoco (Todorov al. 2012). OpenAl Gym (Brockman et al., 2016): Ant, Hopper, HalfCheetah, Humanoid, and Walker2d. To implement sparse reward setting, we adopted the delay method (Oh et al., (2088). We first accumulate extrinsic rewards generated from the considered environments for every A timesteps or until the episode ends. Then, we provide the accumulated sum of rewards to the agent at the end the timesteps or at the end of the episode, and repeat this process. We set A 40 for experiments.
All simulations were conducted over 10 fixed random seeds. The y-axis in each figure with the title "Average Return" represents the mean value of the extrinsic returns of the most recent 100 episodes averaged over the 10 random seeds. Each colored band in figure represents the interval of =o around the mean curve, where is the standard deviation of the 10 instances of data from the 10 random seeds. (Please see Appendix for detailed description of the overall hyperparameters for simulations.)
5

---

Under review as a conference paper at ICLR 2020
3.2 ABLATION STUDY
First, in order to validate the proposed approach in which the intrinsic reward is given by the minimum surprise the ensemble, we investigated several methods of obtaining single intrinsic reward value from the multiple preliminary reward values r iiit:: ... , E,in in (7) from the K models: the proposed minimum selection (12) [13 and other possible methods such the maximum selection, the average value taking method, and pure 1-step surprise method with the mixture with the intrinsic reward defined
Σ 9: P:Pojen (S1+1)5t, Σ 000 oan (8441/8,;
T Tiiine
1-step, en (sc, ae: St41) log
(18)
18
results from the idea that we simply replace the unimodal model Ps in 3 with the mixture
model PK in (
Hopper
HalCCheetah
303
2520
Ce2 mm
K-2. me
6a2
Ksl m7s4
2020
6x2 Leua
1-tap
Anaage Ratu
-20
503
1010
-402
520
-92
6x2 ovg
6-2 1usp
-503
2.5 (Inex(pp)
Imstteppt
Humanoid
K-2 mi
mac
2000
4.0
Weerage Samure
2200
3:00
2000
5:0
K-d,
0.2
0.6
T1mestpp190
Timadeoth
Figure 1: Impact of single intrinsic reward value extraction for K 2: minimum selection (min), maximum selection (max), average (avg), and 1-step surprise with ensemble (I-step).
Fig. shows the mean performance of the four single intrinsic reward value extraction methods for K 2: the proposed minimum selection 13). the maximum selection, the average method, and the pure 1-step surprise with mixture (18). As inferred from Propositions aad it is seen that the minimum selection yields the best performance in all the environments. (The average method yields similar performance in HalfCheetah and Humanoid.) Interestingly, the proposed approach motivated by Propositions/ and outperforms the simple mixture replacement in (18). With this validation, we for all remaining studies.
use the minimum selection method 112 13)
Next, we investigated the impact of the model order K. Since we adopt Gaussian distributions for the dynamics models Pat... ... Pax, the mixture Pк in (4) Gaussian mixture for given state-action pair (s, a). According to recent result (Haarnoja et al. 2018), the model order of Gaussian mixture need not be too large to capture the underlying dynamics effectively in practice. Thus, we evaluated the performance for K = 1,2,3,4. 4.
Fig. 2 shows the mean performance as a function of for the considered sparse reward environments. It is observed that in general the performance improves as K increases, and once the proper model order is reached, the performance does not improve further or degrades a bit due to more difficult model estimation for higher model orders, expected from intuition. From this result, found that = 2 seems reasonable for our model order, so we used K 2 for all the five environments in the following performance comparison 3.3

---

Under review as a conference paper at ICLR 2020
Hopper
HairCheetah
103
2530
-30
-202
593
1000
-32
0
-402
M-1
-503
-502
Timesass(R)
Timsspphi
Humanold
Walker2d
Aueeag Sat.im
2000
-200
2000
0.0
0.2
0.6 Timallest4))
0.0
0.1
Figure 2: Mean performance for considered sparse reward environments function of K. K = 0
means PPO without intrinsic reward, and K means the single- mogle- surprise method. (K =
yielded similar performance to that of K = 3, so we omitted the curve of K for simplicity)
3.3 PERFORMANCE COMPARISON
With the above verification, compared the proposed method with existing intrinsic reward generation methods by using PPO as the background algorithm. We considered the existing intrinsic reward generation methods: curiosity (Pathak et al., 2017). hashing (Tang gain approximation (de Abril & Kanai (2018), and single-model surprise (Achiam & Sastry) (017)). We also considered the method using intrinsic reward module (Zheng al. (018)) among the most recent works introduced in Appendix C which uses delayed sparse reward setup and provides an implementation code.
(017). information
For fair comparison, we used PPO with the same neural network architecture and common hyper- parameters, and applied the same normalization technique in (17) for all the considered intrinsic reward generation methods that the performance difference results only from the intrinsic reward generation method. The weighting factor 3 in (I6) between the extrinsic reward and the intrinsic reward should be determined for all intrinsic reward generation methods. Since each of the considered methods yields different scale of the intrinsic reward, we used an optimized 3 for each algorithm for each environment.
In the case of the single-man modell surprise method and the proposed method, the hyperparameters of the single-model surprise method are tuned to yield best performance and then the proposed method employed the same hyperparameters as the single-model surprise method. We also confirmed that the hyperparameters associated with the other four methods were well-tuned in the original papers (de Abril & Kanai 2018 Pathak et al.) (2017 Tang et al.) 2017 Zheng et al. 2018), 8), and we used the hyperparameters provided by these methods. (Please see Appendix for detailed description of the hyperparameters for simulations.)
Fig. shows the comparison results. It is seen that the proposed reward generation method yields top-level performance. Note that the performance gain by the proposed method significant in sparse Hopper and sparse Walker2d.
intrinsic
RELATED WORK
Various types of intrinsic motivation such as curiosity, information gain, and surprise have been investigated in cognitive science (Oudeyer & Kaplan| 2008), and intrinsically- attiisieally-mooivvaeedd RL has been inspired from these studies. Houthooof et (2016) used the information gain on the dynamics model additional reward based the notion of curiosity. Pathak al. (2017) defined an intrinsic reward
7

---

Under review as a conference paper at ICLR 2020
HairChe
102
2530
-100
1520
-20
Avcrig
-00
1000
503
500
Timesasp(R)
Timaspppii
Timsspphi
Humanoid
Walker2d
5.0-
2500
Teakis
Aveegge APLLI
2000
2200
300
2000
200
per
Cm
5:0
1.00
Bily
0.2
0.6
0.0
0.5
Timostesh))
Figure 3: Performance comparison.
with the prediction error using feature state space, and de Abril Kanai (2018) enhanced Pathak et al. (211))s 's work with the idea of homeostasis in biology. The concept of surprise was exploited to yield intrinsic rewards (Achiam & Sastry 2017).
In parallel with intrinsically motivated RL, researchers developed model-based approaches for learning itself, in which the agent uses the trained dynamics model and fictitious samples generated from the model for training. Nagabandi et al. (2017) suggested using the trained dynamics model to initialize the policy network at the beginning of model-free learning. Kurutach et al. (2018) proposed the policy optimization using trust-region method with model ensemble, in which multiple prediction models for the next state for given pair of current state and action are constructed and trained by using actual samples, and the policy is trained by multiple fictitious sample trajectories from the multiple models. Our work differs from these works in that we use a model ensemble for the environment transition probability distribution and generates intrinsic reward based this ensemble of dynamics models to enhance the performance of model-free RI. with sparse reward. (Please see Appendix [orr more related works.)
CONCLUSION
In this paper, we have proposed a new intrinsic reward generation method based an ensemble of dynamics models for sparse-reward reinforcement learning. In the proposed method, the mixture of multiple dynamics models is used to better approximate the true unknown transition probability, and the intrinsic reward is designed as the minimum of the intrinsic reward computed from each dynamics model to the mixture to capture the most relevant surprise. The proposed intrinsic reward generation method was combined with PPO to construct working algorithm. Ablation study has been performed to investigate the impact of the hyperparameters associated with the proposed ensemble- based intrinsic reward generation. Numerical results show that the proposed mnoell-nssemble-bun based intrinsic reward generation method outperforms major existing intrinsic reward generation methods in the considered sparse environments.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arxiv::/03.0/11 2017.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-5058 2017.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1rxiv:1606.01540 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-drive learning. arXiv preprint arXiv: 04355, 2a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprini arXiv: 1810. 12894, 2018b.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4755-4765, 2018.
Cédric Colas, Olivier Sigaud, and Pierre- Yves Oudeyer. GEP-PG: Decoupling exploration and exploitation in deep reinforcement learning algorithms. arXiv preprini arXiv: (802:0/800200054 2018.
Iidefons Magrans de Abril and Ryota Kanai. Curiosity-driven reinforcement learning with homeostatic regulation. arXiv preprint arXiv: 1000-80110000 1801.07440, 2018.
Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Openai baselines. https://github. com/
Openai/baseling 2017.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329-1338, 2016.
Dror Freirich, Ron Meir, and Aviv Tamar. Distributional multivariate policy evaluation and exploration with the bellman gan. arXiv preprint /00.://80009/66 2018
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01200 2018.
R Devon Hjelm, Alex Fedorov, Samuel Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 1008. 2018
Karan Grewal, Adam Trischler, and
Zhang-War Wei Hong, Tzu- Yun Shann, Shih-Yam Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 10489-10500, 2018
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1 1109-1111 17, 2016.

---

Under review as a conference paper at ICLR 2020
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. EMI: exploration with mutual information. CoRR, abs/1810.01176, 2018. URL http:://axii org/abs/ 1810. 01176
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: /0592, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529, 2015.
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dy- namics for model-based deep reinforcement learning with model-free fine-tuning. arXiv preprint 120Y::7700.02566, 2017.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning, arXiv preprint arXiv: 05635, 2018.
Pierre- Yves Oudeyer and Frédéric Kaplan. How can we define intrinsic motivation? In Proceedings of the 8th International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems, Lund University Cognitive Studies, Lund: LUCS, Brighton. Lund University Cognitive Studies, Lund: LUCS, Brighton, 2008.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), volume 2017.
Alfréd Rényi et al. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium Mathematical Statistics and Probability, Volume I: Contributions to the Theory of Statistics. The Regents of the University of California, 1961.
Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys, Timothy Lilli- crap, and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint arXiv: 110.02481, 2018.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 1506vv5500.00 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Lior Shani, Yonathan Efroni, and Shie Mannor. Exploration conscious reinforcement learning revisited. arXiv preprint / 18/205551, 2018.
Pranav Shyam, Wojciech Jaskowski, and Faustino Gomez, Model-based active exploration. CoRR, abs/1810.12162, 2018
wttp:///.wrrraa//...o..o 12162 12162
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DuTurck, and Pieter Abbeel. # Exploration: A study of count-based exploration for deep reinforeement learning. In Advances in Neural Information Processing Systems, pp.
2750-2759, 2017.
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep rein- forcement learning. In AAAI, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026- 5033. IEEF, 2012.
10

---

Under review as a conference paper at ICLR 2020
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. AAAI, volume 2. pp. 5. Phoenix, AZ, 2016.
Christopher K. I. Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. MIT Press Cambridge, MA, 2006.
Sirui Xie, Junning Huang, Lanxin Lei, Chunxiao Liu, Zheng Ma, Wei Zhang, and Liang Lin. NADPEx: An on-policy temporally consistent exploration method for deep reinforcement learning. arXiv preprint arXiv: 18 182.00008, 2018.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, pp. 4654-4654, 2018.

---

Under review as a conference paper at ICLR 2020
A PROOFS
Proposition Let (T) be the actual expected discounted sum extrinsic rewards defined in
(8
Then, the following inequality holds:
0
< c BiE,a)nr [DKr. ((s)P))((ss))))
(19)
(b)
where c is a positive constant.
Proof. The inequality (a) is trivial from the definition of x², that is, π² is optimal policy maximiz- ing (TT). The inequality (b) holds since
v(mT)) S (7(T)))))))) (..a)<>>' [De (PIIP") a)] since c D, (+1+) 2 0
(20)
(((²)) ) by definition of f((-)
(21)
S 7 by definition #*
(22)
= () c ((s,a)=* [DL (PIIP") by definition of f7(-)
(23)
Proposition 2. Let (-),,)a))) i=..,, K be the ensemble of model distributions, and P(-|s, a) be an arbitrary true transition probability distribution. Then, the minimum of average KLD between P(-|s, a) and the mixture model P' = E, q.P (-]s,, a) over the mixture weights {q..-.- 9K 9 2 E. % 1} upper bounded by the minimum of average KLD between P and Pa. over {i]: ie.
min Beaaa)nx'
Drz
P
Σ PoP
(8,0)
S min E( [DKL (PIIPg.) [(s,a)]
(24)
Proof.
DKL. (p)
Σ diPat [0,0a)
min d: 0, E, 9.
E
P
(25)
B(a.))o Ea.Dae (PIIpg)) (s,
min 0. 9K @ 2 0, Σ,
(26)
min
Σ 9:B(s, [DKL (PIIPg) [(s,a)]
(27)
q @ 2 0. >
min EE (a
[DKL (P[[Py) (ss
(28)
Here, (26) is valid due the convexity of the KL divergence in terms of the second argument for fixed first argument. (27) is valid due to the linearity of expectation. (28) is valid since the minimum in the right-hand side of (27) is achieved when assign all the mass di that has the minimum
value of == [DeL P P (s,, a) (Note that the optimal {qc} in (27) is not the same as the
optimal {4: achieving the minimum in 25
Note that each step in the proof is tight except (6) in which the convexity of the KL divergence in terms of the second argument is used. This part involves the function f(x) = log æ for < x <
since DRr(pilles (1p2) f pı (v) log pı(y) dy, but the convexity of f(x) -log for < x < is
not severe if is not close
12

---

Under review as a conference paper at ICLR 2020
B NEURAL NETWORK ARCHITECTURE AND HYPERPARAMETERS
2017 7) is used. The policy For the actual implementation, the code implemented by Dhariwal et al.
and dynamics models were designed by fully-connected neural networks all of which had two hidden layer of size (64, 64) (Dhariwal et al. [017 Houthooft et al. 2016 Tang et al. 2017)) The tanh activation function was used for all of the networks (Achiam & Sastry, 2017 7 Dhariwal et al. 2017). The means of the fully factorized Gaussian dynamics models were the outputs of our networks, and the variances were trainable variables which were initialized
(Dhariwal et al.. 2017). Other than the variances, all initialization is randomized so that each of dynamics models was set differently (Kurutach et al. [2018: Tavakoli et al. 2018). For the implementation of the policy model, our method and all the considered intrinsic reward generation method used the same code for the module method (Zheng et al. 2018).
Although recent work (Achiam & Sastry 2017 used TRPO (Schulman et al. 2015a) the baseline learning engine, we used PPO (Schulman et al. 2017). ) one of the currently most popular algorithms for continuous action control, as baseline a algorithm. While the same basic hyperparameters as those in the previous work (Achiam & Sastry| 2017) were used, some hyperparameters were tuned for PPO. X for the GAE method (Schulman al. (015b) was fixed to 0.95, while the discounting factor was set to 7 = 0.99. The batch size L for the training of the policy was fixed to 2048. For the policy update using PPO, the minibatch size Limie was set 64, the epoch number N 10, the clipping constant 0.2, and the entropy coefficient 0.0. The maximum number of timesteps was 1M for all five environments, and the maximum index of batch period, M AX," is [T] 488. The learning late of Adam optimizer (Kingma& & Ba 2014) was fixed to 0.0003.
Each of the single-moder surprise method, the hashing method and our proposed method requires replay buffer. The size of the used replay buffer for all these three methods is I.. Before the beginning of the iterations, 2048 x B samples from real trajectories generated by the initial policy were added to the replay buffer. We set B = 40 for our experiments. For the methods not requiring replay buffer, i.c., Curiosity, Information Gain, Module, and PPO Only, we ran 2048 x B = 81920 timesteps before measuring performance for fair comparison. (Therefore, every x-axis in Fig. and 3 shows the total timesteps of 1.08 (081.MM.)
For the dynamics model learning, set the batch size L' 2048, L Lmming = 64, and N' 4. The optimization (14) was solved based on second-order approximation (Schulman et al.) 2015a). When
K= 1, the optimization (14 reduces the model learning problem in Achiam & Sastry (2017). In
Achiam & Sastry 2017), 7), the constraint constant i: in the second-order optimization was well-tuned as 0.001. So, we used this value of K not only to the case of K 1 but also the case of K 2 2. We further tuned the value of a in (14) for each environment, and we set a = 0.01. For the information gain method, we need another hyperparameter / which the weight to balance the original intrinsic reward and the homeostatic regulation term (de Abril & Kanai 2018). We tuned this hyperparameter for each environment and the used value of h shown in T
T summarrzes the weighting factor 3 as well as the hyperparameter h in information gain method. Here, we used the optimized weighting factor /8 in (6) for each algorithm for each environment. As aforementioned, the major hyperparameters for the proposed model-ensemble method are the same as those used for the single-model surprise method.
Ant
Hopper
HalfCheetah
Humanoid
Walker2d
Curiosity Hashing
B = 0.01
8 = 0.01
8 = 0.0001
B = 0.1
3 == 0.03
3 0.0001
8 = 0.01
8 = 0.00001
8 0.1
8 0.003
Information Gain
$ = 0.01. h =
4
8 3= 0.1, = 4
8 = 0.0001. h = 4
B 3= 0.01, h 2
3 3= 0.03, h h= 2
Single Surprise
3 0.00001
8 0.05
8 0.0001
8 0.3
g 3= 0.05
Table 1: Used hyperparameter values.
For the intrinsic reward module method, we checked that the provided source code in github repro- duced results in Zheng et al. (2018), as shown in Fig. 4 'Module 0.01" represents the module method with training using the sum of intrinsic reward and scaled extrinsic reward with scaling factor 0.01. 'Module O' represents training using intrinsic reward only (no addition of extrinsic reward). Both methods are introduced in Zheng al. (2018), and we checked reproducibility when B = 0, i.e., we
13

---

Under review as a conference paper at ICLR 2020
2048 B 0 timesteps before measuring performance. We observed that our used code yielded the same results as those in Zheng et al. (2018).
Ant 0)
Hopper (8 = 0)
HalfCheetah (# 0)
Moque.0
Nadui 000
-102
2020
1003
-202
1000
503
520-
-102
Mosake
0 01
Moesk, 0.11
Walker2d
5.0-
2500
Mid.ds,5
4.00
Aeweage Reture
2000
8:0
2000
Makuh Maauh
Timastssh0
Figure 4: Reproduced mean performance the module method over 10 random seeds with A 40
when B = O.
Thus, we used this code for the module method with only one change that 2048 B timesteps with B = 40 before measuring performance for fair comparison. (Since the range of intrinsic reward from the module method is [-1, 1]. intrinsic reward normalization in (77 is needed.) For the module method in performance comparison 3.3 we selected better method between 'Module O' and 'Module 0.01', assuming B 40. 'Module performed better than 'Module 0.01* Hopper and Walker2d, and 'Module 0.01" performed better than 'Module O' in the other three environments.
14

---

Under review as a conference paper at ICLR 2020
C MORE RELATED WORK
Recent advanced exploration methods can be classified mainly into two categories. One is to generate intrinsic reward explicitly and train the agent with the total reward which is the sum of the extrinsic reward and the adequately scaled intrinsic reward. The other is indirect methods which do not explicitly generate intrinsic reward. Our work belongs the first category. There exist many exploration techniques on image spaces (Bellemare et al. 20166] Burda et al. 2018a 2018) but these works are not directly related to our work here.
Savinov et al.
1. Explicit Intrinsic Reward Generation
Andrychowicz al. 2 suggested a new intrinsic reward for sparse and binary extrinsic reward environments, based on sampling additional states from the replay buffer and setting those data as new goals. In their work the policy was based on the input of both state and goal. In our work, on the other hand, the concept of goal is not necessary. A recent work by Zheng et al. (2018) used delayed reward environment to propose training the module to generate intrinsic reward apart from training the usual policy. This delayed reward environment for sparse reward setting is different from the previous sparse reward environment based on thresholding (Houthooft et al.) 2016), i.c., the agent get non-zero reward when the agent achieves certain physical quantity (such the distance from the origin) larger than the predefined threshold. Recently, |Freirich et al. (2018) proposed generating Recently,,Freirih
intrinsic reward by applying generative model with the Wasserstein- distance. With the concept divergence (JSD) (Hjelm) al. 2018 construct new variational lower bound of the corresponding mutual information, guaranteeing numerical stability. Our work differs from these two recent works in that we used a model ensemble generate intrinsic reward.
of state-action embedding, Kim et al. (2018) adopted the
2. Exploration without Intrinsic Reward Generation
Recent indirect methods can further be classified mainly into two groups: (i) revising the original objective function to stimulate exploration, which exploits intrinsic motivation implicitly, and (ii) perturbing the parameter space of policy.
In the first group. Oh et al. (2018) proposed that exploration can be stimulated by exploiting novel state-action pairs from the past, and used sparse reward environments by delaying extrinsic rewards. Hong et al. (2018) revised the original objective function for training by considering maximization of the divergence between the current policy and recent policies, with an adaptive scaling technique. The concept of dropout was applied to the PPO algorithm to encourage the stochastic behavior of the agent episode-wisg wisely (Xie et al. 2018). Convex combination of the target policy and any given policy is considered a surrogate Markov Decision Process, generalizing usual exploration methods such as é-greedy or Gaussian noise.
exploratory policy (Shani et al. 2018), which corresponds to solving
In the second group. Colas et al. 2018 proposed goal-based exploration method for continuous control, which alternates generating parameter-outcome pair and perturbing certain parameters based on randomly drawn goal from the outcome space. Recently, inspired by Chua et al.) (2018), |Shyam al. (2018) considered pure exploration MDP without any extrinsic reward with the notion of utility, Jensen-Rényí
where utility is based on JSD and the Jensen- Rényi divergence (Rényi et al., 1961). In this work, they consider a number of models for transition function but they used this to compute utility based on average entropy of the multiple models. Our work uses the minimum of surprise from the multiple dynamics models under the existence of explicit reward whether it extrinsic or intrinsic.
15


Reviews:
[{"review_id": "ryefwM-CtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes an auxiliary reward for model-based reinforcement learning. The proposed method uses ensembles to build a better model of environment dynamics and suggests some rules to optimize the new ensemble-based dynamics and to estimate the intrinsic reward.\n\nI am torn on this paper. I like the derivation of the method and the ideas behind it. I think it is an interesting direction of research. However, the experiments are limited to one domain and the paper needs proofreading. I will vote \"weak accept\" for this paper, as I think it is incremental and the experiments are too limited.\n\nAs I said above, the paper could use some proofreading. Some sections (like pages 1-2) are well written, but others are full of grammatical mistakes. There is also a lot of redundant information.\n\nIt is often stated that the ensemble model has better capacity than the single model. Some experimental proof of this better modelling capacity could help convince a reader that the ensemble is indeed beneficial and warranted (e.g., show that P_K is better than the single model P).\n\nSome evaluation or discussion on the computational costs of the method would be beneficial. I assume the ensemble-based method is more computationally intensive. Would it perform better than single surprise if they were compared according to wall clock time?\n\n\nExamples of minor issues:\n\n- Page 3, after equation 7, sentence beginning with \"In addition to the advantage that the mixture model (4) has the increased model order for modelling\" is confusing, contains redundant elements, and is not bringing useful information. It should be revised\n- page 4, in paragraph after eq. 11: the following sentence is grammatically incorrect, please revise: \"Propositions 1 and 2 suggest a way to intrinsic reward design under the mixture model for tight gap between η(π ̃∗) and η(π∗)\"\n- in the same sentence, revise: \"be close to the true η(π∗) of the true optimal policy π∗, as desired.\",\n- page 6 text above Figure 1: \"single-modal\" should be unimodal\n\n\nPOST REBUTTAL \n\nThanks for writing your rebuttal. I have read it, as well as the other reviews. I think reviewer 1 touches on some important points, especially regarding the engineered sparse rewards. It seems the method is not properly justified given the environments used for its evaluation. Based on this, and the fact that the method is rather incremental, I would like to change my score to a weak reject. The method should be evaluated in a setting­ with truly sparse rewards.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJeUWy3iYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach for using an ensemble of learning dynamics models to generate an intrinsic reward for reinforcement learning in sparse reward environments. The paper's results demonstrate that this approach out-performs other benchmark approaches across a set of continuous control tasks.\n\nI'm curious about the motivation for taking the min surprise across the ensemble. Eqs 8-11 seem to motivate that this will make a tighter gap between the expected cumulative returns of the optimal policy and the policy with additional intrinsic rewards. This implies that this will converge to the same policy in the end, but I'm not sure that it implies that this is a good exploration strategy, an intrinsic reward of 0 would provide an even tighter bound. When you used the max or average return, was the problem that there was still too much intrinsic reward for the agent to converge? Or that it wasn't exploring enough or in the right places?\n\nFor the experiments, it would be very interesting to see the intrinsic rewards accumulated over time for each approach. That plot could be very enlightening as to what is going on. \n\nIt would also be great to see the results on these tasks when they're not modified to be sparse reward. Is your method a big hindrance in that case? Or does it still help?\n\nYour intro claims that typical model-free RL is about the circumstance where the agent receives non-zero reward for every time step. This is not true, many model-free RL papers look at tasks that have sparse reward on some or many steps.  I would agree that in many continuous control problems, shaping rewards are added to ease the exploration problem. \n\nFor Figure 3, how does the model-ensemble TRPO (Kurutach et al) fit in? Is that algorithm represented by one of the curves?\n\nHere's one more related work deriving intrinsic rewards from an ensemble of trained dynamics models:\nhttp://www.sciencedirect.com/science/article/pii/S0004370215000764", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJepVJVitS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a model-based method for intrinsic rewards based on probabilistic neural network ensembles. For a particular ensemble element, an intrinsic reward is defined as a log-term that measures the deviation in prediction between a Gaussian mixture over all ensemble elements and the particular ensemble element (at the previous update period). The intrinsic reward signal applied in practice is the minimum of the aforementioned quantity w.r.t. all ensemble elements. The authors provide some theoretical motivation for their approach and validate their method in sparse-reward continuous robotics domains (Mujoco), using PPO as reference algorithm. Experiments are averaged over 10 seeds and compared against various other intrinsic reward baselines including PPO without intrinsic motivation. In these experiments, the method provided by the authors seems to dominate over competing approaches.\n\nQuality\n\nI find the quality low. First, while I appreciate that the authors try to provide some theoretical motivation for their method, there is quite a gap from theory to practice. The theory assumes that the world model is known and considers a stationary reward signal. In practice, the world model is approximated with the current network ensemble, a quantity that is changing over time yielding a non-stationary signal. Second, the experiments are conducted in a sparse-reward modification of Mujoco-type environments that are non-sparse by construction. Sparsity is introduced by accumulating instantaneous rewards over some time window before \"releasing\" them. This way of sparsifying rewards yields weird partial observability issues, e.g. the same state-action pair observed at the right moment in time yields significant reward whereas at the wrong moment in time yields no reward at all. Additionally, it is always guaranteed that there will be a reward signal at a certain frequency. There are probably better environments for studying the approach, like Atari for example. I do understand that the authors cite Oh et al. 2018 who apply the same technique of sparsification, but Oh et al. also conduct additional experiments in ALE.\n\nClarity\n\nThe paper is clearly written and easy to follow. On a side note, it could be stated more clearly that the presented approach is not model-based because no forward prediction is required for constructing intrinsic rewards (merely probability values are queried for observed transitions). There is one question I have though regarding the experiments where different intrinsic reward approaches are compared against each other. The paper states that all approaches normalize intrinsic rewards according to Equation (17)---why do all of them then need a different weighting factor \\beta as stated in the second paragraph of Section 3.3? Furthermore, since \\beta is fine-tuned for each intrinsic reward approach and each environment, can the authors please elaborate in detail how exactly this is accomplished (to ensure correct interpretability of the results)?\n\nOriginality\n\nThe originality is low. As stated by the authors, the proposed method is an incremental extension of Achiam and Sastry 2017 who proposed a similar method for non-ensemble methods.\n\nSignificance\n\nThe significance is low as well. The method's improvement over other intrinsic reward approaches is minor (the environments chosen by the authors are also not ideal). I feel the significance is reduced further by the fact that there are other model-based approaches that use models of the proposed kind for increasing sample-efficiency and performance considerably in non-sparse environments (e.g. Kurutach et al.2018).\n\nUpdate\n\nAfter reading the other reviews and the rebuttal, the concerns I have raised remain. I still feel this paper shouldn't be accepted to ICLR. However, given the extensive experimental analysis conducted by the authors, I feel a score of 1 from my side was too harsh. I therefore increase to 3.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the challenge of sparse reward reinforcement learning through intrinsic reward generation based on the deviation in predictions of an ensemble of dynamics models. This is combined with PPO and evaluated in some Mujoco domains.

The main issue here was with the way the sparse rewards were provided in the experiments, which was artificial and could lead to a number of problems with the reward structure and partial observability. The work was also considered incremental in its novelty. These concerns were not adequately rebutted, and so as it stands this paper should be rejected.
Decision: Reject