Risk Averse Value Expansion for Sample Efficient and Robust Policy Learning
Authors: Bo Zhou
Fan Wang
Hongsheng Zeng
Hao Tian
Venue:  (2020)
Keywords: reinforcement learning
model-based RL
risk-sensitive
sample efficiency

Abstract: Model-based Reinforcement Learning(RL) has shown great advantage in sample-efficiency, but suffers from poor asymptotic performance and high inference cost. A promising direction is to combine model-based reinforcement learning with model-free reinforcement learning, such as model-based value expansion(MVE). However, the previous methods do not take into account the stochastic character of the environment, thus still suffers from higher function approximation errors. As a result, they tend to fall behind the best model-free algorithms in some challenging scenarios. We propose a novel Hybrid-RL method, which is developed from MVE, namely the Risk Averse Value Expansion(RAVE). In the proposed method, we use an ensemble of probabilistic models for environment modeling to generate imaginative rollouts, based on which we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on different environments including MuJoCo and robo-school show that RAVE yields state-of-the-art performance. Also we found that it greatly prevented some catastrophic consequences such as falling down and thus reduced the variance of the rewards.

Under review as a conference paper at ICLR 2020
RISK AVERSE VALUE EXPANSION FOR SAMPLE EFFI- CIENT AND ROBUST POLICY LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Model-based Reinforcement Learning(RL) has shown great advantage in sample- efficiency, but suffers from poor asymptotic performance and high inference cost. A promising direction is to combine model-based reinforcement learning with model-free reinforcement learning, such as model-based value expansion(MVEE) However, the previous methods do not take into account the stochastic character the environment, thus still suffers from higher function approximation errors. As result, they tend to fall behind the best model-free algorithms in some chal- lenging scenarios. We propose a novel Hybrid-RL method, which developed from MVE, namely the Risk Averse Value (xpansion(RAVE) In the proposed method, we use an ensemble of probabilistic models for environment modeling to generate imaginative rollouts, based which further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on different environments including MuJoCo and robo-school show that RAVE yields state-of-the-an performance. Also we found that it greatly prevented some catastrophic consequences such as falling down and thus reduced the variance of the rewards.
INTRODUCTION
In contrast to the tremendous progress made by model-free reinforcement learning algorithms in the domain of games(Mnih has risen up as great challenge to RI., especially when interacting with the real world. Toward this challenge, a promising direction of the learning process (Sutton process(Sutton 1991 Calandra et al.) 2016 Kalweit & Boedecker, 2017; 7; Oh et al.)
al.] 2015 Silver et al.) [017]] Vinyals al., (2019), poor sample efficiency
integrate the dynamics model to enhance the sample efficiency
2017 Racanière et al. 2017). However, classic model-based reinforcement learning(MBRR)) meth- ods tend lag behind the model-free methods(MFRI asymptotically, especially in cases of noisy environments and long trajectories. The hybrid combination MFRL and (HybrHy short) has attracted much attention due this reason. A lot of efforts has been devoted to this field, including the dyna algorithm( (uuruthh(Kuruttc Kurutach et al. 2018). model-based value (Feinbeenn et al. 2018). 12A Weber et al. 2017), etc.
for
The robustness of the learned policy is another concern in RL. For stochastic environments, pol- icy can be vulnerable tiny disturbance and occasionally drop into catastrophic consequences. In MFRL, off-policy RL(such as DQN, DDPG) typically suffers from such problems, which in the end leads instability in the performance including sudden drop in the rewards. To solve such problem, risk sensitive MFRL not only maximize the expected return, but also try to reduce those catastrophic outcomes/Garcia & Fernández, 2013) Dabney et al., 2018a] Pan al. 2019). For MBRL and Hybrid-RI. without modeling the uncertainty in the environmenti espeecally for contin- uous states and actions), often leads to higher function approximation errors and poorer perfor- mances. It is proposed that complete modeling of uncertainty in transition can obviously improve the performaace et al. 2018), however, reducing risks MBRL and Hybrid-R -L has not been sufficiently studied yet.
In order to achieve both sample efficiency and robustness at the same time, we propose a new Hybrid-RL method more capable of solving stochastic and risky environments. The proposed method, namely Risk Averse Value Expansion(RAVEs is an extension of the model-basee value ex- Dansion(MVE al. 2018) and stochastic ensemble value (STAnii

---

Under review as a conference paper at ICLR 2020
et al. 2018). We systematically analyse the approximation errors of different methods in stochastic environments. We borrow ideas from the uncertainty modeling( Chua et al. (2018)) and risk averse reinforcement learning. The probabilistic ensemble environment model is used, which captures not only the variance in estimation(also called epistemic uncertainty), but also stochastic transition na- ture of the environment(also called aleatoric uncertainty). Utilizing the ensemble of estimations, we further adopt dynamic confidence lower bound of the target value function to make the policy more risk-sensi We compare RAVE with prior MFRL and Hybrid-RL baselines, showing that RAVE not only yields SOTA expected performance, but also facilitates the robustness of the policy.
RELATED WORKS
The model-based value expansion(MVB//Feinbeeg et al. 2018) is a Hybrid-RL algorithm. Un- like typical MFRL such as DQN that uses only I step bootstrapping, MVE uses the imagination rollouts of length H predict the target value. The assistance of environment model can greatly improve the sample efficiency at the start, but the precision of long term inference becomes limited asymptotically, In order to properly balance the contribution of the value expansion of different horizons, stochastic ensemble value expansion(STEVEL (Bucka)a tion of value expansion of different horizon. The accuracy of the expansion is estimated through the ensemble of environment models as well as value functions. Ensemble of environment models also models the uncertainty to some extent, however, ensemble of deterministic model captures mainly epistemic uncertainty instead of stochastic transitions Chua et al. 2018).
(Buckman et al. 2018) adopts an interpola-
The uncertainty or the function approximation error is typically divided into three (lassssGeman ett al., 1992): the noise exists in the objective environment, e.g., the stochastic transitions, which is also called alcatoric uncertainty(Chun et al., 2018). The model bias is the error produced by the limited expressive power of the approximating function, which is measured by the expectation of ground truth and the prediction of the model, in case that infinite training data is provided. The variance is the uncertainty brought by insufficient training data, which is also called epistemic un- certainty. Dabney et al.] (2018b) discuss the epistemic and aleatoric uncertainty in their work and focus on the latter one to improve the distributional RL. Recent work suggests that ensemble of probabilistic model(PE) is considered as more thorough modeling of uncertainty(Chua et al.] [2018), while simply aggregate deterministic model captures only variance or epistemic uncertainty. The stochastic transition is more related to the noise(or aleatoric uncertainty), and the epistemic uncer- tainty usually of interest to many works terms of sxploitation&eexplraaiionoo
et al. 2017 Schmidhuber 2010 Oudeyer & Kaplan| 2009). Other works adopt ensemble of deterministic value function for exploration(Osban et al. 2016 Buckman et al. 2018).
Risks in RL typically refer to the inherent uncertainty of the environment and the fact that policy may perform poorly in some cases(Garcia & Fernández) 2015). Risk sensitive learning requires not only maximization of expected rewards, but also lower variances and risks in performance. To- ward this object, some works adopt the variance of the return (Sato et al.] [2001] Pan et al.] 2019] Reddy et al. 2019), the worst-case Matcome(Heeeen 1994 Gaskett] 2003) in either policy learn- ingnPan et al. 2019 Reddy et al., (2019). exploration(Smine (mminnova et al. 2019), or distributional value (sainey
al.J 220 8a). An interesting issue in risk reduction that reduction of risks is typically found to be conflicting with exploration and exploitation that try to maximize the reward in the long run. Authors in (Pan al. 2019) introduce two adversarial agents(risk aversion and long- term reward seeking) that act in combination to solve this problem. Still, it remains quite tricky and and risk seeksseeng/xpploratiiio in RL. In this paper, we
empiristic to trade-off between propose a dynamic confidence bound for this purpose.
number of prior works have studied function approximation error that leads to overestimation and sub-optimal solution in MFRL. Double DQN(DDQN) Van Hasselt et al. 2016) improves over DQN through disentangling the target value function and the target policy that pursues maximum value. In (Fujimooo et al., 2018) the authors suggest that systematic overestimation of value function also exists in actor-critic MFRL. They use an ensemble of two value functions, with the minimum estimate being used as the target value. Selecting the lower value estimation is similar to using uncertainty or lower confidence bound which is adopted by the other risk sensitive methods(Pan et al. 2019), though they claimed different motivations.
2

---

Under review as a conference paper at ICLR 2020
3 PRELIMINARIES
3.1 ACTOR-CRITIC MODEL-FREE REINFORCHMENT LEARNING
The Markov Decision Pro) is used describe the process of agent interacting with the environment. The agent selects the action and a feedback rt E from the environment. As
A at each time step t. After executing the
action, it receives a new observation St-1 we focus mainly the environments of continuous action, we denote the parametric deterministic policy that the agent uses to decide its action as a; - [to())). Typically we add Gaussian exploration noise on top of the deterministic policy, such that have stochastic behavioral policy Ta,a S + R. is calculated as To.- az) = (N o²), where (a) o²) represents the probability density at x in Gaussian distribution (m,, o²). As the interaction process continues, the agent generates a trajectory 7 (sy, ag Fo: $1, 01: (0.:'1..... following the policy ~0.0. For finite horizon MDP, we use the indicator d S + {0, 1 to mark whether the episode is terminated. The objective of RL is to find the optimal policy π to maximize the expected discounted sum of rewards along the trajectory. The value performing the action with policy at the state is defined by Er= 7'ri, where y is the discount factor. The value iteration in model-free RL tries to approximate the optimal value Q with parametric value function Q by minimizing the Temporal Difference(TD) error, where ø the parameter be optimized. The TD eπror between the estimates of Q-value and the corresponding target values is shown in equation. []] where ø' delayed copy of the parameter &, and a' ww, with O' being a delayed copy of A(Lillicrap et al. 2015).
Q (s,a) = =, a,,
C = E, (i 8441) - (o((),
(I)
with Qrarger (Te, Se+1) =r + (1- Qu
To optimize the deterministic policy function in continuous action space, deep deterministic policy (DDG)) et al.] 2015) maximizes the value function (or minimizes the negative value function) under the policy Me with respect to parameter 0, shown in equation. 2
C = EE, E Qui (er,
(2)
3.2 ENVIRONMENT MODELING
To model the environment continuous space, an environment model typically composed of three individual mapping functions: fr.c. S x x + R. f..c. S x A A-5 S, and fá.ca S - [0, 1]. which used to approximate the feedback, next state and probability of the terminal indicator respectively(Gu et al., 2016; Feinberg et al., 2018). Here G. So and Ça are used to represent the parameters of the corresponding mapping functions. With the environment model, starting from 8t, at, we can predict the next state and reward by
8 fa.c, (St, oz), fr.g, (St, a, 14+1), fa.ca (8++11)
(3)
and this process might go on to generate a complete imagined trajectory of [st, Pr, $141.....
The neural network commonly used as an environment model due its powerful express ability. To optimize the parameter Ç we need to minimize the mean square error(or the cross entropy) of the prediction and the ground truth, given the trajectories under the behavioral policy.
3.3 UNCERTAINTY AWARE PREDICTION
The deterministic model approximates the expectation only. As mentioned in the previous sec- tions, overestimation is attributed to the error in function approximation. Following Chua et al. 2018), briefly review different uncertainty modeling techniques.
Probabilistic models output a distribution (c.g., mean and variance of a Gaussian distribution) in- stead of an expectation. We take the reward component of the environment model example,
3

---

Under review as a conference paper at ICLR 2020
the probabilistic model is written as N(frr... O....). and the loss function is the negative log likelihood(equation.
-.
C By [ogpw(rilf:s, ae 8111), (st as S(+1))
(4)
Ensemble of deterministic(DE) models maintains ensemble of parameters, which is typically trained with different training samples and different initial parameters. E.g. given the ensemble of parameters G,,1, Sr...... .... Sr.N, the expectation and the variance of the prediction is acquired from
equation. G
1 Es, [frss.] N
Ere.. Vs. [frs.] Eiirs.. Bsz, [frss.)
(5)
We define the operator E.g and V, the average and the variance on the ensemble of æes respec- tively. As proposed by (Chua et al. 2018). the variance o² in equation. 4 mainly captures the aleatoric uncertainty, and the variance V mainly captures the epistemic uncertainty.
Ensemble of probabilistic models(PE) keeps track an collection of distributions ( o "r.,,.)) ),. [I, which can further give the estimation of both uncertainties. A sampling form PE goes as follow
Sample unformlly from N], then sample r ~ (frrrrra P...n.))
(6)
3.4 MODEL-BASED VALUE EXPANSION
In MVEneein et al. 2018 the learned environment model fcrcc.sa = (face fr.G.: fa.ca) to-
gether with the policy ss used to image trajectory starting from state 8, and action a, which represented by Cc..Co.Sa.0' (Tt, 81+1). It produces a trajectory up to horizon H(H 2 0). We can
write Ca H (Tt, St+1) = (Te, St41, â,+1, fr41, $142. H++ h++
Then the target value Qtarget equation. replaced with estimated return êVE the sampled trajectory Sa..5.. (Tt, 8441), which is expressed in equation
Ca- on
Qtarget (re, 8141) +
ÔMVE
Grtnnaal ".n.....nnn wn (Te, S(41)
H+
Σ
H+ x "y + + d 1 0+
=
(7)
6' II k=t+2
with de,r (1 d(se41))
- (ac
3.5 STOCHASTIC ENSEMBLE VALUE EXPANSION
Selecting proper horizon // for value expansion important to achieve high sample efficiency and asymptotic accuracy at the same time. Though the increase of H brings increasing prophecy, the asymptotic accuracy is sacrificed due the increasing reliance on the environment model. of different
In STEVE Buckman et al. 2018), interpolating the value expansions C-SS..ca.0re H. [0, Hmaz] is proposed. The weight for each horizon is decided by the inverse its variance, which is calculated by estimating ensemble of values switching the combination of parameters in environment model f and value function Qø. Through notation, STEVE be written as equation. g
Σlmor wH BuB.e.c.cadd
(QMVE Qarta.a. ca & w.w... (r1,
STEEVE (Te,st11)
Σ ΣH WH
Hmax
with wH = Ve..orssid AMVE of (re, (441)
4

---

Under review as a conference paper at ICLR 2020
4 INVESTIGATION OF THE APPROXIMATION ERROR IN STOCHASTIC ENVIRONMENTS
To thoroughly investigate the impact of aleatoric uncertainty on hybri-RL. methods, we construct a demonstrative toy environment(fn. [(a)). The agent starts from So = 0, chooses an action ae from A = [-1, +1] at each time step L. The transition of the environment is 8141 and 1, where k represents the deterministic transition, and 1 represents the stochastic transition. The episode terminates ([s) > 5 ), where the agent acquires a final reward. The agent gets a constant penalty(-100) at each time step to encourage it to reach the terminal state soon possible. Note that the deterministic environment actually requires more steps expectation to reach |s > 5| compared
+ oel +
(0, 1). We compare two different environments.k
with the stochastic environment, thus the value function at the starting point of k = 1 (Ground truth 380+) tends to be lower than that of k = 0(Ground truth = 430+).
Left
Right
So
-5
-2
(a) Toy Environment
Uncerteinty)
EnvrromeettLet))
500
Estimatec Q
Estimated
400
STEVE NVE DDPG Ground-Truth
STEVE
MVE DDPG
Drmund-Truth
200
700
Steps
(b) The change of ( au 1) with k
(c) The change of ( au 1) with k
Figure I: Curves of estimated Q values in the toy environment at the starting point the environ- ment Each experiment run four times.
We apply the learning methods including DDPG, MVE, STEVE to this environment, and plot the changes of estimate values at the starting point(see fig. (5)
The results show that, in the deterministic environment, the Q-values estimated by all methods converge to the ground-truth asymptotically in such simple environment. However, after adding the noise, previous MFRL and Hybrid-RL methods show various level of overestimation. The au- thors of (Feinberg et al., 2018) have claimed that value expansion improves the quality of esti- mated values, but MVE and STEVE actually give even worse prediction than model-free DDPG in the stochastic environment. A potential explanation is that the overall overestimation comes from the unavoidable imprecision of the estimator( Thrun & Schwartz [993] Fujimoto et al.) 2018), but Hybrid-Re - also suffers from the approximation error of the dynamics model. When using de- terministic environment model, the predictive transition of both environments would be identical, because the deterministic dynamics model tends approximate the expectation of next states(e.g.
fa.s, (s, = 0, a > 0) = 1.0, fo.c, (s, = 1.0, a, > 0) = 2.0). This would result in the same value es- timation for & 0 and & = 1 for both value expansion methods, but the ground truth of Q-values different these two environments. As result, the deterministic environment introduces additional approximation error, leading more severe overestimation.
5

---

Under review as a conference paper at ICLR 2020
5 METHODOLOGY
5.1 RISK AVERSE VALUE EXPANSION
We proposed mainly two improvements based on MVE and STEVE. Firstly, we apply an ensemble of probabilistic models (PE) to enable the environment model to capture the uncertainty over possi- ble trajectories. Secondly, inspired by risk sensitive RL, we calculate the confidence lower bound of the target value(fig lcc).
Figure casual illustration of three Distributional Value Expansion(DVE)) trajectories.
Before introducing RAVE we start with the Distributional Value Expansion(DVE), Compared with MVE that uses a deterministic environment model and value function, DVE uses a probabilistic en- vironment model, and we independently sample the reward and the next state using the probabilistic environment models(see equation and fig. 2)
8142 N(fo,s (86+1, uet1 uo (1(+1)), 0..s. (s1-1, U(+1))
Fet1 ~ N (S (81+1, (4,1, 8t42), on, (8e+1, 0411,
(9)
d(E+42) N ((f.ccc od. (3x42))
We apply the distributional expansion starting from Te, S++1 acquire the trajectory Ca (rt, $1+1) (11,8141, 14411,441. 2e.,...., $444+1, 0++ +1), based on which we write
DVE as equation. 10
Cr (Te,
Σ
2 2' de + + ++ dr, (G
=
8=t+1
(10)
with de (1 d(se+1))
II (1 d(Bx)) k=t+2
We then keep track of an ensemble the combination of the parameters (Gr, Cos Sd. d'}. For each group of parameters we use an ensemble of N parameters, which gives 4N parameters in all. Then we select random combination of four integers {i,j. k, I}. which gives the parameter com- bination of {Sr.ss Sa.gs Sd (1) By switching the combination of integers we acquire ensemble of DVE estimation. Then we count the average and the variance on the ensemble of DVE, and by subtracting certain proportion(a) of the standard variance, we acquire lower bound of DVE esti- mation. We call this estimation of value function the a-confidence lower bound(a-CI.) written as equation. [1
Q Qa-CLB
CLB (Te.8t41) = EC Caidad [OVEE Sa. or H. (Te, sc+1)
(11)
a Vensensais
(OVE P M. (Te, s(+1)
Subtraction variances is commonly used in sessitensi RL Kobayashi 2000 Pan et al. 2019 Reddy al. 2019). The motivation is straight forward, try to suppress the utility of the

---

Under review as a conference paper at ICLR 2020
high- high-variance trajectories, in order to avoid possible risks. However, the here is left undecided. We will come to this problem in the next part.
Finally, define RAVE, which adopts the similar interpolation among different horizons STEVE based on DVE and CLB, shown equation.
13
E -H=O wn CLBCLB CLB
Uu (re, 86+1)
Qtarget S441)
QRAVE (142))42))
LH=0 WH
(12)
with WH = Gaiisiaaad Criaaaaa' (Tr, ((+1)
While adopting the lower confidence bound may introduce the bias of underestimation, it makes the policy less preferable to actions with large variance of future return.
5.2 ADAPTIVE CONFIDENCE BOUND
An unsolved problem in RAVE is to select proper a. The requirement of risk aversion and ex- ploration is somehow competing: risk aversion seek to minimize the variance, while exploration searches states with higher variance. To satisfy both requirements, previous work proposed two competing agents, and each will make decision for a short amount of time et al. 2019)). Here we propose another solution to this problem. We argue that the agent needs to aggressively explore at the beginning, and it should get more risk sensitive as the model converges. A key indicator of this is the epistemic uncertainty. The epistemic uncertainty measures how well our model get to know the state space. In MBRL and Hybrid-RL, there is common technique to easily monitor the epistemic uncertainty, by evaluating the ability of the learned environment model predict the consequence of its own actions(Pathak et al. 2017).
Following this motivation, we set the confidence bound factor to be related to its current state and action, denoted as a (s,, a). We want a(s, a) to be larger when the environment model could perfectly predict the state to get more risk sensitive, and smaller when the prediction is noisy to allow more exploration. We have
ass,, 04) = mar
| (f (s)) st-1l??),
(13)
where Z is scaling factor for the prediction error. With a little abuse of notations, we use here to represent a constant hyperparameter, and ass,a) is the factor that is actually used in a-CLB. a(s a(st, ae) picks the value near zero at first, and gradually increases with the learning process.
EXPERIMENTS AND ANALYSIS
We evaluate RAVE continuous control environments using the MuJoCo physics simula- torTTooorov et al., 2012). The baselines includes the model-free DDPG and STEVE that cur- rently yields the SOTA Hybrid-RL performance MuloCo. We also align our performance with the SOTA MFRL methods including twin delayed deep deterministic (TD3) policy gradient algo- rithn(Fujimoto et al. 2018 soft actror-critic(SAC
algorithm(HHarnoja al. 2018), and proximal policy optimization( PPO) Scchulman et al.) (017). using the implementation provided by the au- thors. To further demonstrate the robustness in complex environments, we also evaluate RAVE on OpenAl's Roboschool (Klimov & Schulman 2017). where STEVE has shown large improvement than the other baselines. We detail hyper-parameters and the implementation in the supplementary materials.
6.1 EXPERIMENTAL RESULTS
We carried out experiments eight environments shown in fig. 3 Among the compared methods, PPO is the only on-policy updating method, which has very poor sample-efficiency compared with either STEVE off- ooflpollcy MFRL methods, as PPO needs large batch size to learn stably (aarnojj ctt al. 2018). DDPG achieves quite high performance in HalfCheetab-vl and Swimmer-V but fails almost all the other environments, especially challenging environments such Humanoid.

---

Under review as a conference paper at ICLR 2020
HalfCheetah-vl
Swimmer-vv
Hopper -v
4300
000
Score:
Score
1000
Envirenment Steps
Environment Steps
Steps
Walker2d.v2
Humanoid
BipeddlWWllerrllardccreeey
Score
Score
Score
200
263
3,100
2001
Enviranment Steps
Environment Steps
Env Environmnnt Steps
RobosshholHHmaandd--y1l
RebosshhalluumandddFianu-x-
RAVE (H=3)
Score
Scora
STEVE (H-3)
3X0
DDPG
PPO
TD3
$3.60
536
SAC
Enviranment Steps
Environment Steps
Figure 3: Average return over environment frames in MuJoCo and Roboschool environments. Each
experiment is run four times.
In Hopper-v1 and Walker2d-v1, STEVE not compare with TD3 and SAC, which yields quite good performance in many environments. However, RAVE performed favorably in most environ- ments in both asymptotic performance and the rising mmeadnme sample efficiency), except for HalfCheetah-vl and Swimmer-vl, where DDPG has already achieved satisfying performance and the margin between DDPG and Hybrid-RL. is not that large.
6.2 ANALYSIS
Distribution of Value Function Approximation. To investigate whether the proposed method predicts value function more precisely, we plot the of the predicted values Q against the ground truth values of Hopper-vl in fig. + The ground truth value here is calculated by directly adding the rewards of the left trajectory, thus it is more like a monte carlo sampling from ground truth distribution, which quite noisy. To better demonstrate the distribution of points, we draw the confidence ellipses representing the density. The points are extracted from the model at environment steps of IM. In DDPG and STEVE, the predicted value and ground truth aligned poorly with the ground truth, while RAVE yields better alignments, though a little bit of underestimation.
DOPG
STEVE-DOPG
RAVE-DOPG
0.8
8.6
04
nomma InTUTTO
Figure 4: Confidence ellipses of the distribution of estimated values Q versus the ground truth in Hopper-v1. The points extracted from environment steps of 1M, each includes statistics from 10,000 points. The x-axis and y-axis represent the statistical cumulative discounted returns(ground truth) and the predictive Q-values(both normalized), respectively.
8

---

Under review as a conference paper at ICLR 2020
Investigation dynamic confidence bound. In order study the role played by the a-confidence lower bound separately, we further carried out series of ablative experiments in Hopper-vl environ- ment. We compare RÁVE(a =constant), RAVE(dynamic a(s, a)) and other baselines in chart. For all the algorithms, we set Maax = 3, and the experiments are replicated for 4 times.
0(which means ensemble of DVE only) already surpasses the performance of STEVE in Hopper-vl v1, showing that modeling aleatoric uncertainty through PE indeed benefits the performance of value expansion. Larger margin is attained by introducing a- CLB. A very large assuch as constant a = 2.0, which means lower CLB) can quickly stabilize the performance, but its performance stayed low due to lack of exploration. while a smaller accnstaant = 0.0, 0.5 generates larger fluctuation in performance. The dynamic adjustment of a facilitates quick rise and stable performance.
From fig. 5(a) can that
Hopper-u1
Humanodivv1
BAT
E0PG R51 (a=3]
RAT 11-1
163
3020
0.
500e
2020
1020
02
STEVE
TB3
D0PG
SAC
100300
200020
30100C
430000
500000
200030
400000
800030
1000000
Enwirormen Stops
mmont 504ps
(a) Performance of RAVE variants
(b) Falling rate
Figure 5: Examining the performance of RAVE variants and robustness of the learned policy. Each experiment is run four times. (a) Performance of RAVE with different a-CLB.(b) The falling rate of
various algorithms over the first 1M environment steps. Each point is evaluated on the outcome of 1000 episodes.
Analysis on Robustness. We also investigate the robustness of RAVE and baselines on the most challenging Mujoco environment, Humanoid-v1. Humanoid-vl involve complex humanoid dynam- ics, where the agent is prone to fall. We evaluate the robustness with the possibility of falling by the learned policy, As shown in fig 5(b) RAVE achieves the lowest falling rate compared with the baselines.
Computational Complexity. A main concern toward RAVE may be its computational complexity. On the one hand, hybrid-RL involves the usage of environment model which introduces additional computational cost. On the other hand, RAVE and STEVE involves ensemble of trajectory rollouts, which is a little bit costly. We keep the ensemble size the same as STEVE and the details about the hyper-parameter can be found in the supplements.
For the training stages, the additional training cost of RAVE compared with STEVE comes from modeling aleatoric uncertainty and additional sampling cost. We tested the training speed of STEVE and RAVE, and the time for RAVE to finish training 500 batches with batch size 512 is 13.20s, an increase of 24.29%, compared to STEVE(10..22 GPUs with 8 CP0s(2.00GHZ). For the inference stages, RAVE charges exactly the same computa- tional resources just as the other model-free actor critic methods as long as the model architecture of the policy is equal, which is lot more cost efficient compared with MBRL that adopts planning procedure.
The time reported here is tested in 2 P40 Nvidia
Also want to emphasize here that the computation complexity is typically less important com- pared with sample efficiency, as the interaction cost matters more than computational cost in training procedure.

---

Under review as a conference paper at ICLR 2020
7 CONCLUSION
In this paper, we raise the problem of incomplete modeling of uncertainty and insufficient robustness in model-based value expansion. We introduce ensemble probabilistic models approximate the environment, based on which we introduce the distributional value expansion(DVEL) a-Confidence Lower Bound, which further leads RAVE. Our experiments demonstrate the superiority of RAVE in both sample efficiency and robustness, compared with state-of-ther art RL methods, including the freee--free TD3 algorithm and the Hybrid-RL STEVE algorithm. We hope that this algorithm will facilitate the application of reinforcement learning in real-world, complex and risky scenarios.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances Neural Information Processing Systems, pp. 8224 8234, 2018.
Roberto Calandra, André Seyfarth, Jan Peters, and Mare P. Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence (AMAI), 76 (1):5-23, 2016. ISSN 1573-7470. doi: 10.1007/10447--055-966-999
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754-4765 2018.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. arXiv preprint arXiv:1806.0668 2018a.
Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018b.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101, 2018
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. 2018.
Javier Gareia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480) 2015. 2015.
Chris Gaskett. Reinforcement learning under circumstances beyond its control. 2003.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma.
Neural Computation, 4-1): 1992. doi: 10.1162//eco..1192..11 URL /tttts::/dll
1992..11...
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference Machine Learning, pp. 2829- 2838, 2016
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01299 2018.
Matthias Heger. Consideration of risk in reinforcement learning. In Machine Learning Proceedings 1994, pp. 105-111. Elsevier, 1994.
Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven imagination for continuous deep rein- forcement learning. In Conference on Robot Learning, pp. 195-206, 2017.. 2017.
O. Klimov and J. Schulman. Roboschool. httpar//ggittub..coo///eea////se 2017.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint urXiv:/802. 10592, 2018.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (509v0000090977 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):52,, 2015.
11

---

Under review as a conference paper at ICLR 2020
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed frame- work for emerging (AI) applications. In 13th {USENIX) Symposium on Operating Systems Design and Implementation ((OSDI) 18), pp. 561-577, 2018.
Junhyuk Oh, Satinder Singh. and Honglak Lee. Value prediction network. In Advances in Neural Information Processing Systems, pp. 61 1118-6288, 2017.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-4034, 2016.
Pierre-Yvel Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computa-
tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforcement learning. arXiv preprint arXiv: 1904.0051 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. imagination-augmented agents for deep reinforcement learning. In Advances in neural information processing systems, pp. 5690-5701, 2017.
D Reddy, Amrita Saha, Srikanth G Tamilselvam, Priyanka Agrawal, and Pankaj Dayama. Risk averse reinforcement learning for mixed multi-agent environments. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2171-2173. Inter- national Foundation for Autonomous Agents and Multiagent Systems, 2019.
Makoto Sato and Shigenobu Kobayashi. Gariance-penalizer reinforcement learning for risk-averse asset allocation. In International Conference on Intelligent Data Engineering and Automated Learning, pp. 244-249. Springer, 2000.
Makoto Sato, Hajime Kimura, and Shibenobu Kobayashi. Td algorithm for the variance of return and mean-variance reinforcement learning. Transactions of the Japanese Suciety for Artificial Intelligence, 16(3):353-32, 2001.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 233):20- 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
David Silver. Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature. 550(7676):35,, 2017.
Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforrettt learning. arXiv preprint arXiv: 0ttp:::992007000000 2019.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160-163, 1991.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, 1993.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
12

---

Under review as a conference paper at ICLR 2020
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathicu, Max Jaderberg, Woj- ciech M. Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dal- ibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Yuhuai Wu, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. https: htttps://deepmiine /ttp//eeeeednnnee/////e//m com/blog/
Siphastar-masteriie
real-time strategy game starcraft- 2019.
Théophane Weber, Sébastien Racanière, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, al. imagination-augmenttd agents for deep reinforcement learning. arXiv preprint 2017.
13

---

Under review as a conference paper at ICLR 2020
A TRAINING AND IMPLEMENTATION DETAILS
A.1 NEURAL NETWORK STRUCTURE
We used rectified linear units (ReLUs) between all hidden layers of all our implemented algorithms.
Unless otherwise stated, all the output layers of model have no activation function.
RL Models. We implement model-based algorithms on top of DDPG, with policy network and Q-value network. The policy network is stack of 4 (FFleecconect)d((C)) layers. The activation function of the output layer is tanh constrain the output range of the network. The Q-value network takes the concatenation of the state and the action
input, followed by four FC
layers.
Dynamics Models. We train three neural networks as the transition function, the reward function and the termination function. We build eight FC layers for the transition approximator, and four FC layers for the other approximators. The distributional models Wf.. σ²) in RAVE use the similar model structure except that there two output layers corresponding the mean and the variances respectively.
A.2 PARALLEL TRAINING
We distributed training accelerate our algorithms and the baseline algorithms. Following the implementation of STEVE, we train a GPU learner with multiple actors deployed in CPU cluster. The actors reload the parameters periodically from the learner, generate trajectories and send the trajectories to the learner. The learner stores them in the replay buffer, and updates the parameters with data randomly sampled from the buffer. For the network communication, we use Ray(Moritz) et al. 2018) to transfer data and parameters between the actors and the learner. We have 8 actors generating data, and deploy the learner on the GPUs. DDPG uses GPU, and model-based methods uses two: one for the training of the policy and another for the dynamics model.
80
Transition Function
Q-value Function
Reward Function
Cartesian Product
Value Expansion
Figure 6: An illustration of the rollout result with N 4, Hmar = 3, For each state of the trajectory, there 16 candidate targets.
A.3 ROLLOUT DETAILS
We employ the identical method of target candidates computation STEVE, except we im- age rollout with an ensemble of probabilistic models At first, we bind the parameters of transition model(çs,)) to the termination model((a.). That is, we numerate the combina- tion of three integers {i,j. j,i,j,k k E [1], which gives an ensemble of 3 parameters (Gris Sa.is Čd,i o }. The actual sampling process goes like this: For each H E [0, Hmar), we first use the transition model(ç..:) and the termination model(Ça,) image state-action sequence (st-1, 0z 8,42, 2r+2.... ..., 2+4H41 ax+ 1 }; Based on the state-action sequence, we use reward function({r.g) estimate the rewards (fr. frisng (Se, 8e41)) and the value function((')) to predict the value of the last state(Q (1 a++ ()ig. 60. In total we predict N (Hmax + 1) combination of rewards and value functions in both RAVE andSTEN
14

---

Under review as a conference paper at ICLR 2020
B HYPER-PARAMETERS FOR TRAINING
We list all the hyper-parameters used in our experiments in table
Table I: Table of hyper-parameters
Hyper-parameter
Value
Description
Batch size for training the RI., and also the dynamics model
B
512
N Nrpm
le6
Size of the replay buffer storing the transitions
[r
3e-4
Learning rate of the training policy
IrQ
3e-4
Learning rate of the Q-value function
lrp
3e-4
Learning rate of the dynamics model
Probability of adding Gaussian noise to the
0.05
action for exploration
H Mmaz
Maximum horizon length for value expansion
Ensemble size of the value function and
environment models
Number of collected frames to pretrain the dynamics model before training the policy Scaling factor for the prediction error Confidence lower bound in equation 13
F
10000
Z
alpha
1.5
15


Reviews:
[{"review_id": "SylI6P9iqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper expands on previous work on hybrid model-based and model-free reinforcement learning. Specifically, it expands on the ideas in Model-based Value Expansion (MVE) and Stochastic Ensemble Value Expansion (STEVE) with a dynamically-scaled variance bias term to increase risk aversion over the course of learning, which the authors call Risk Averse Value Expansion (RAVE). Experimental results indicate notable improvements over their selected model-free and hybrid RL baselines on continuous control tasks in terms of initial learning efficiency (how many environment steps are needed to achieve a particular level of performance), asymptotic performance (how high the performance is given the same large number of environment steps), and avoidance of negative outcomes (how infrequently major negative outcomes are encountered over the course of training).\n\nReview\nThe core contribution of the paper is an extension to STEVE that uses an idea from risk-averse RL of biasing the underlying estimators away from high-variance predictions, and adds a dynamic weight to that bias.\n\nStrengths\n- The addition of a risk-aversion term to STEVE is a good contribution to the literature on safe RL. While it may be possible to criticize this contribution as somewhat trivial, I am disinclined to do so, as finding simple ideas that are effective is the hallmark of an important contribution, in my opinion.\n- Under the assumption that the baselines are fair, the empirical results show substantial improvements for a good selection of challenging tasks along three metrics: initial efficiency, asymptotic performance, and avoidance of negative outcomes.\n- The paper provides a careful and detailed section on the relevant preliminaries, giving precise notation that is expanded step-by-step from basic actor-critic approaches all the way through the description of RAVE. (Although see my comment below about possibly making the notation more concise.)\n- Overall, the paper’s presentation is clear and natural. (Although see my comment below about a strong need for proofreading.)\n\nWeaknesses\n- The paper opens with a claim that MBRL has worse performance than MFRL in environments with “noisy environments and long trajectories”. However, recent work [1] has shown that MBRL can outperform MFRL on many of the tasks considered in this paper using far fewer environment steps, even when training directly from pixels.\n- The experiments are all based on observations of the true state vectors rather than pixels, which dramatically simplifies working with imagination rollouts, as the dimensionality is so much lower. This hides the high computational cost and modeling challenges involved in using imagination-based MBRL or hybrid RL approaches in real-world environments, and (in my opinion) is a major shortcoming of this line of research (not only of this paper). Since the DDPG family of algorithms doesn’t have to do rollouts, they have a strong advantage over this type of approach on more realistic settings where the true state is unknown and must be inferred from high-dimensional observations. If experiments are not going to be done using pixels, the discussion should at least mention the trade-offs involved between the proposed algorithm and the baselines in that setting.\n- The core baseline of the paper is DDPG, which is unnecessarily weak. D4PG [2] came out a year and a half ago and has the same high-level properties as DDPG, but outperforms it on all of the Mujoco tasks considered in this paper. Additionally, it includes some of the ideas presented in this work, but in a model-free setting, including a distributional treatment and multi-step rollouts, so it is easy to imagine that some of the experimental gains presented here would be erased when using it as a baseline.\n- The use of ensembling is another confounder in the experiments. Ensembling models almost always yields an improvement, so any technique that relies on some form of ensembling needs to additionally demonstrate that the gains presented are not solely due to ensembling. In this case, the experiments with STEVE and RAVE should minimally be performed with different numbers of models in their ensembles.\n- (Minor) Similarly, the prediction horizon can have a large impact, and only STEVE and RAVE consider prediction horizons other than 1. Showing how the horizon affects performance would help make the comparisons more fair.\n- Figure 1 shows a bias problem in some current approaches in a clear toy problem, but does not show whether RAVE addresses that problem. Even though this section precedes the presentation of RAVE, it is necessary to show that the proposed solution actually helps on the toy problem.\n- There are a number of problems with the proposal for adaptively computing the alpha parameter. In general, such adaptive hyperparameters add a great deal of complexity to hyperparameter tuning, so such suggestions should be either strongly motivated by theory or by empirical results. Neither seems to be the case here.\n  - Only one proposal for adaptive alpha (equation 13) is considered. Its justification is plausible, but it would be more convincing if other dynamic approaches were considered in the experiments. For example:\n    - alpha(env_step) = alpha * env_step / max_env_step\n    - alpha(s_t,a_t) = alpha * min{1.0, 1 / (Z * ||E[fhat(s_t,a_t)] - s_t+1||^2)}\n    - The opposite of the proposed approach, where alpha starts high and gets lower over the course of training.\n  - Indeed, a potentially useful quantity during evaluation, when s_t+1 is unknown, would be based on the variance of the predicted next state, rather than the difference of the expectation of the predicted next state and the true next state. E.g.:\n    - alpha(s_t,a_t) = alpha / Var[fhat(s_t,a_t)]\n    This formulation says that the precision of the prediction determines the confidence of the model, which is also intuitively reasonable (to me, anyway) and doesn’t rely on knowing the future.\n  - (Minor) The adaptive variant appears to be only slightly better than alpha=0.5, but requires two hyperparameters that are unspecified in the main body of the paper — alpha and Z. The appendix lists those parameters, but no discussion is made on how much tuning was done to determine that pair of parameters.\n  - (Minor) A plot of how the dynamic value of alpha changes during training would be useful, at least in the appendix.\n\nRecommendation\nIn light of my comments above, I cannot currently recommend the acceptance of this paper at ICLR. However, I think that the core idea is likely to hold up under more careful experimental comparisons. If the authors submit a revised draft that addresses the substance of my concerns, I would be very likely to increase my rating. In particular, I would like to see much more careful experimental treatment of the idea, so that readers can have high confidence about the circumstances where RAVE is likely to be a good choice.\n\n[1] Hafner et al., “Learning Latent Dynics for Planning from Pixels”. ICML 2019. https://arxiv.org/abs/1811.04551\n[2] Barth-Maron et al., “Distributed Distributional Deterministic Policy Gradients”. ICLR 2018. https://arxiv.org/abs/1804.08617\n\n\nOther Comments and Questions\n- This paper needs a lot of proofreading. A few examples of errors that should be fixed before publication:\n  - “equation. 2”: This should read “Equation 2”, “equation 2”, “Eq. 2” or “eq. 2”. The period in the last two options indicates that letters have been elided. The mistake of using a period where no elision has happened occurs throughout the paper.\n  - “prophesy”: This is not the correct word. Just say “prediction”.\n  - “image”: This is often used when the correct word would be either “imagine” or “imagination”.\n  There are many other errors that could be fixed easily with the help of a native English speaker.\n- The mathematical notation in sections 3 through 5 is precise, but it is also a bit heavy. Consider whether there would be any ambiguity added if, for example, {\\hat Q}^DVE_{\\zeta_s,\\zeta_r,\\zeta_d,\\theta’,\\phi’} were instead notated {\\hat Q}^DVE_{\\zeta,\\theta’,\\phi’}.\n- Figure 2 adds nothing of value to the paper and should be removed.\n- Consider comparing on DMControl, which is the same set of tasks as Mujoco, but the scores are standardized such that each task has a maximum reward of 1000 per episode.\n- Why does DDPG get worse over time on the RoboSchool tasks? Without a clear explanation, it looks like a bug, and a bug like that calls into question the rest of the DDPG results as well.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Ske-fbzGqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of high function approximation errors facing the stochastic environments when trying to combining model-based reinforcement learning (RL) with model-free RL. The paper begins by showing that previous methods like model-based value expansion (MVE) and stochastic ensemble value expansion (STEVE) can perform even worse than the pure model-free DDPG algorithm in a rather noisy environment. Following the ideas of MVE and STEVE, It then proposes a risk averse value expansion (RAVE) to replace the target Q function in the actor-critic algorithm, which is built upon an ensemble of probabilistic models (PE) and adopt the lower confidence bound as a surrogate of the target value as in the risk-sensitive RL. A simple yet intuitive approach for adaptively selecting the confidence bound \\alpha is also proposed. The experiments show that RAVE does improve over the state-of-the-art algorithms in several different environments, with a better draw-down control. In general, this paper is well-written and the idea of RAVE is novel as far as I know. But since I'm not very familiar with the specific literature of combing model-based and model-free RL, and since the idea of RAVE is relatively straightforward (but admittedly practically powerful and theoretically interesting), I choose to give a conservative accept to account the possibility that some existing works have followed very similar approaches. \n\nSome minor comments: 1) What is the loss of the negative log-likelihood in (4) for? 2) The authors may want to explain clearly what is the variance indicating in (8) more clearly, although one can guess that it is closely related to (5). ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1etZbgy9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work , the authors combine model-based value expansion(MVE) with\nmodel-free reinforcement learning and also take into account the high-order stochastic characteristics\nof the environment to make the value expansion algorithm in modeling error propagation of dynamics risk-averse. \nThey propose a novel Hybrid-RL method, namely the Risk Averse Value Expansion(RAVE), that uses an ensemble of probabilistic dynamics models to generate imaginative rollouts and to model risk aversion\nof risks by estimating the lower confidence bound of the ensemble. They showed that by taking the risk (in terms of variance) of the dynamic propagation error into the account, RAVE can achieve comparable performance with other state-of-the-art baselines in benchmark experiments including MuJoCo and robo-school. Also they showed that RAVE reduced the variance of the return and thus prevented catastrophic consequences such as falling.\n\nI found this work interesting as the authors try to take the uncertainty of probabilistic dynamics model into account for estimating the value function and its confidence bounds in model-free RL. Utilizing such confidence interval of values, they can have a way of doing exploration while being risk-averse. While similar approaches of modeling the mean and variance of Q functions can be found in some existing work, such as boot-strapped DQN (Osband'16), and efficient exploration via Bayesian DQN (Azizzadenesheli'18), none of these work model the variance of the Q-function using the error propagation of the dynamics model. Through extensive experiments, it has also demonstrated the effectiveness of RAVE in terms of achieving good performance, approximation error in value estimation, as well as robustness to failure in standard mujoco benchmarks, which provides readers some detailed understanding on how this risk-averse uncertainty modeling/propagation technique helps in exploration. However, while i found this idea interesting, it appears to me that the current work is still quite preliminary (without theoretical concentration inequality based variance bounds for guiding exploration), and empirically, it would be great to compare this method with the aforementioned bootstrapping/bayesian based approaches. My other major comment is on claiming the proposed method to be \"risk-averse\", because in RL, risk-averse methods are commonly known to not only optimize the expected return but also to provide guarantees to other moments of the return, such as variance of CVaR. However, while the method used variance of the expected value (due to the error propagation of the dynamics) for exploration, I am not seeing the risk-averse optimization criteria being studied here. Therefore calling the RL method risk-averse might be a little mis-leading.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bylh6JURtH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel deep reinforcement learning algorithm at the intersection of model-based and model-free reinforcement learning: Risk Averse Value Expansion (RAVE). Overall, this work represents a significant but incremental step forwards for this \"hybrid\"-RL class of algorithms. However, the paper itself has significant weaknesses in its writing, analysis, and presentation of ideas.\n\nThe main strength of this work is its empirical section. The proposed algorithm is fairly compared against many relevant baselines on a variety of continuous control tasks, on both the Mujoco and Roboschool simulators, and demonstrates significant performance increases across all of them. The visualization in Figure 4 is interesting, and provides good insight into the issues with DDPG and STEVE, and the reasons for the success of RAME. Based on these results, the authors have convinced that RAME is a state-of-the-art algorithm.\n\nHowever, in spite of its strong performance on benchmarks, I believe that this paper needs a significant overhaul/rewrite before publication.\n\nOne major criticism I have is on the authors' treatment of the different types of uncertainty. The introductory sections spend a large amount of time on the differences between aleatoric and epistemic uncertainty, and various other related concepts. But when it comes to the actual new algorithm, that the authors only barely touched upon the (very important) point that V[Q^DVE] is actually a *mix* of epistemic and aleatoric uncertainties. When minimizing this term, it's not clear whether epistemic or aleatoric uncertainty is actually being reduced. (Based on the experiments, which report only expected values, not CVaR or anything of the sort, it seems the authors care only about reducing epistemic uncertainty; if so, it's unclear why they choose to minimize a term which includes aleatoric uncertainty too.) A more principled understanding of this quantity seems essential to this line of work. Similarly, \"risk\" typically refers to aleatoric uncertainty, but the risk-senstive component of RAME computes the confidence lower bound w.r.t. a mix of aleatoric and epistemic uncertainty. Calling this algorithm \"risk sensitive\" is likely to generate confusion, in my opinion.\n\nA few other points on presentation of ideas:\n- Switching from a deterministic to a stochastic model is a trivial extension of MVE/STEVE, and way too much time is spent on this point (even going so far as to name a new algorithm!). Section 4 contains no new insight, beyond \"deterministic models can be bad in stochastic environments\", which is obvious. Consider re-thinking your experiments for this section to help readers better understand *what* goes wrong.\n- In my opinion, RAME is as much a successor to TD3 as it is to STEVE. Taking the lower outcome of an ensemble of size 2 is equivalent to applying a penalty based on the stdev of the outcome distribution, with α=1. The introduction of the paper should be re-written to clearly point out that that RAME ~= TD3 + STEVE.\n- I'd also like to see a lot more ablations, on at least a few environments. There are at least four factors that need to be teased apart: 1) deterministic vs stochastic model 2) lower confidence bound penalty 3) adaptive α for LCBP 4) STEVE reweighting. Does RAME require all of these elements to perform well? How do these elements interact?\n\nSome feedback on the writing:\n- There are various small factual errors. For example, in the very first paragraph, the Dyna algorithm is attributed to Kuturach et al, instead of Sutton (1990). (The algorithm from Kuturach is ME-TRPO.)\n- The notation is not very good; there are lots of symbols flying around everywhere, and it makes the ideas (which are fundamentally very simple) a bit difficult to parse. For example, naming each parameter-set individually, everywhere, is unnecessary.\n- There are some strange non-sequiturs and overall lack of flow.\n\nI think that this work has a lot of potential, and am especially impressed by the empirical results. I recommend rejection in its current form, but hope to see a revised version of this work appear at a conference in the near future.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to extend model-based/model-free hybrid methods (e.g., MVE, STEVE) to stochastic environments. They use an ensemble of probabilistic models to model the environment and use a lower confidence bound of the estimate to avoid risk. They found that their proposed method yields state-of-the-art performance over previous methods.

The valid concerns by Reviewers 1 & 4 were not addressed by the authors and although the authors responded to Reviewer 3, they did not revise the paper to address their concerns. The ideas and results in this paper are interesting, but without addressing the valid concerns raised by reviewers, I cannot recommend acceptance.
Decision: Reject