Year: 2019

Abstract:
Model-based Reinforcement Learning(RL) has shown great advantage in sampleefficiency, but suffers from poor asymptotic performance and high inference cost. A promising direction is to combine model-based reinforcement learning with model-free reinforcement learning, such as model-based value expansion(MVE). However, the previous methods do not take into account the stochastic character of the environment, thus still suffers from higher function approximation errors. As a result, they tend to fall behind the best model-free algorithms in some challenging scenarios. We propose a novel Hybrid-RL method, which is developed from MVE, namely the Risk Averse Value Expansion(RAVE). In the proposed method, we use an ensemble of probabilistic models for environment modeling to generate imaginative rollouts, based on which we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on different environments including MuJoCo and robo-school show that RAVE yields state-of-the-art performance. Also we found that it greatly prevented some catastrophic consequences such as falling down and thus reduced the variance of the rewards.

heading: 1 INTRODUCTION
text: In contrast to the tremendous progress made by model-free reinforcement learning algorithms in the domain of games(Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019), poor sample efficiency has risen up as a great challenge to RL, especially when interacting with the real world. Toward this challenge, a promising direction is to integrate the dynamics model to enhance the sample efficiency of the learning process(Sutton, 1991; Calandra et al., 2016; Kalweit & Boedecker, 2017; Oh et al., 2017; Racanière et al., 2017). However, classic model-based reinforcement learning(MBRL) methods tend to lag behind the model-free methods(MFRL) asymptotically, especially in cases of noisy environments and long trajectories. The hybrid combination of MFRL and MBRL(Hybrid-RL for short) has attracted much attention due to this reason. A lot of efforts has been devoted to this field, including the dyna algorithm(Kurutach et al., 2018), model-based value expansion(Feinberg et al., 2018), I2A(Weber et al., 2017), etc.
The robustness of the learned policy is another concern in RL. For stochastic environments, policy can be vulnerable to tiny disturbance and occasionally drop into catastrophic consequences. In MFRL, off-policy RL(such as DQN, DDPG) typically suffers from such problems, which in the end leads to instability in the performance including sudden drop in the rewards. To solve such problem, risk sensitive MFRL not only maximize the expected return, but also try to reduce those catastrophic outcomes(Garcıa & Fernández, 2015; Dabney et al., 2018a; Pan et al., 2019). For MBRL and Hybrid-RL, without modeling the uncertainty in the environment(especially for continuous states and actions), it often leads to higher function approximation errors and poorer performances. It is proposed that complete modeling of uncertainty in transition can obviously improve the performance(Chua et al., 2018), however, reducing risks in MBRL and Hybrid-RL has not been sufficiently studied yet.
In order to achieve both sample efficiency and robustness at the same time, we propose a new Hybrid-RL method more capable of solving stochastic and risky environments. The proposed method, namely Risk Averse Value Expansion(RAVE), is an extension of the model-based value expansion(MVE)(Feinberg et al., 2018) and stochastic ensemble value expansion(STEVE)(Buckman
et al., 2018). We systematically analyse the approximation errors of different methods in stochastic environments. We borrow ideas from the uncertainty modeling( Chua et al. (2018)) and risk averse reinforcement learning. The probabilistic ensemble environment model is used, which captures not only the variance in estimation(also called epistemic uncertainty), but also stochastic transition nature of the environment(also called aleatoric uncertainty). Utilizing the ensemble of estimations, we further adopt a dynamic confidence lower bound of the target value function to make the policy more risk-sensitive. We compare RAVE with prior MFRL and Hybrid-RL baselines, showing that RAVE not only yields SOTA expected performance, but also facilitates the robustness of the policy.

heading: 2 RELATED WORKS
text: The model-based value expansion(MVE)(Feinberg et al., 2018) is a Hybrid-RL algorithm. Unlike typical MFRL such as DQN that uses only 1 step bootstrapping, MVE uses the imagination rollouts of length H to predict the target value. The assistance of environment model can greatly improve the sample efficiency at the start, but the precision of long term inference becomes limited asymptotically. In order to properly balance the contribution of the value expansion of different horizons, stochastic ensemble value expansion(STEVE)(Buckman et al., 2018) adopts an interpolation of value expansion of different horizon. The accuracy of the expansion is estimated through the ensemble of environment models as well as value functions. Ensemble of environment models also models the uncertainty to some extent, however, ensemble of deterministic model captures mainly epistemic uncertainty instead of stochastic transitions(Chua et al., 2018).
The uncertainty or the function approximation error is typically divided into three classes(Geman et al., 1992): the noise exists in the objective environment, e.g., the stochastic transitions, which is also called aleatoric uncertainty(Chua et al., 2018). The model bias is the error produced by the limited expressive power of the approximating function, which is measured by the expectation of ground truth and the prediction of the model, in case that infinite training data is provided. The variance is the uncertainty brought by insufficient training data, which is also called epistemic uncertainty. Dabney et al. (2018b) discuss the epistemic and aleatoric uncertainty in their work and focus on the latter one to improve the distributional RL. Recent work suggests that ensemble of probabilistic model(PE) is considered as more thorough modeling of uncertainty(Chua et al., 2018), while simply aggregate deterministic model captures only variance or epistemic uncertainty. The stochastic transition is more related to the noise(or aleatoric uncertainty), and the epistemic uncertainty is usually of interest to many works in terms of exploitation&exploration(Pathak et al., 2017; Schmidhuber, 2010; Oudeyer & Kaplan, 2009). Other works adopt ensemble of deterministic value function for exploration(Osband et al., 2016; Buckman et al., 2018).
Risks in RL typically refer to the inherent uncertainty of the environment and the fact that policy may perform poorly in some cases(Garcıa & Fernández, 2015). Risk sensitive learning requires not only maximization of expected rewards, but also lower variances and risks in performance. Toward this object, some works adopt the variance of the return(Sato et al., 2001; Pan et al., 2019; Reddy et al., 2019), or the worst-case outcome(Heger, 1994; Gaskett, 2003) in either policy learning(Pan et al., 2019; Reddy et al., 2019), exploration(Smirnova et al., 2019), or distributional value estimates(Dabney et al., 2018a). An interesting issue in risk reduction is that reduction of risks is typically found to be conflicting with exploration and exploitation that try to maximize the reward in the long run. Authors in (Pan et al., 2019) introduce two adversarial agents(risk aversion and longterm reward seeking) that act in combination to solve this problem. Still, it remains quite tricky and empiristic to trade-off between risk-sensitive and risk-seeking(exploration) in RL. In this paper, we propose a dynamic confidence bound for this purpose.
A number of prior works have studied function approximation error that leads to overestimation and sub-optimal solution in MFRL. Double DQN(DDQN)(Van Hasselt et al., 2016) improves over DQN through disentangling the target value function and the target policy that pursues maximum value. In TD3(Fujimoto et al., 2018) the authors suggest that systematic overestimation of value function also exists in actor-critic MFRL. They use an ensemble of two value functions, with the minimum estimate being used as the target value. Selecting the lower value estimation is similar to using uncertainty or lower confidence bound which is adopted by the other risk sensitive methods(Pan et al., 2019), though they claimed different motivations.

heading: 3 PRELIMINARIES

heading: 3.1 ACTOR-CRITIC MODEL-FREE REINFORCEMENT LEARNING
text: The Markov Decision Processes(MDP) is used to describe the process of an agent interacting with the environment. The agent selects the action at ∈ A at each time step t. After executing the action, it receives a new observation st+1 ∈ S and a feedback rt ∈ R from the environment. As we focus mainly on the environments of continuous action, we denote the parametric deterministic policy that the agent uses to decide its action as at = µθ(st). Typically we add Gaussian exploration noise on top of the deterministic policy, such that we have a stochastic behavioral policy πθ,σ : S×A → R. It is calculated as πθ,σ(st, at) = pN (at|µθ(st), σ2), where pN (x|m,σ2) represents the probability density at x in a Gaussian distribution N (m,σ2). As the interaction process continues, the agent generates a trajectory τ = (s0, a0, r0, s1, a1, r1, ...) following the policy πθ,σ. For finite horizon MDP, we use the indicator d : S → {0, 1} to mark whether the episode is terminated. The objective of RL is to find the optimal policy π∗ to maximize the expected discounted sum of rewards along the trajectory. The value performing the action a with policy π at the state s is defined by Qπ(s, a) = Es0=s,a0=a,τ∼π ∑∞ t=0 γ
trt, where 0 < γ < 1 is the discount factor. The value iteration in model-free RL tries to approximate the optimal valueQπ ∗ with a parametric value function Q̂φ by minimizing the Temporal Difference(TD) error, where φ is the parameter to be optimized. The TD error between the estimates of Q-value and the corresponding target values is shown in equation. 1, where φ′ is a delayed copy of the parameter φ, and a′ ∼ πθ′ , with θ′ being a delayed copy of θ(Lillicrap et al., 2015).
L φ = Eτ [∑ t (Q̂target(rt, st+1)− Q̂φ(st, at))2 ]
with Q̂target(rt, st+1) = rt + γ · (1− d(st+1)) · Q̂φ′(st+1, a′))
(1)
To optimize the deterministic policy function in a continuous action space, deep deterministic policy gradient(DDPG)(Lillicrap et al., 2015) maximizes the value function (or minimizes the negative value function) under the policy µθ with respect to parameter θ, shown in equation. 2.
L θ = −Eτ [ ∑ t Q̂φ′(st, µθ(st))] (2)

heading: 3.2 ENVIRONMENT MODELING
text: To model the environment in continuous space, an environment model is typically composed of three individual mapping functions: f̂r,ζr : S × A× S → R, f̂s,ζs : S × A → S , and f̂d,ζd : S → [0, 1], which are used to approximate the feedback, next state and probability of the terminal indicator respectively(Gu et al., 2016; Feinberg et al., 2018). Here ζr, ζs and ζd are used to represent the parameters of the corresponding mapping functions. With the environment model, starting from st, at, we can predict the next state and reward by
ŝt+1 = f̂s,ζs(st, at), r̂t = f̂r,ζr (st, at, ŝt+1), d̂t+1 = f̂d,ζd(ŝt+1), (3)
and this process might go on to generate a complete imagined trajectory of [st, at, r̂t, ŝt+1, ...].
The neural network is commonly used as an environment model due to its powerful express ability. To optimize the parameter ζ we need to minimize the mean square error(or the cross entropy) of the prediction and the ground truth, given the trajectories τ under the behavioral policy.

heading: 3.3 UNCERTAINTY AWARE PREDICTION
text: The deterministic model approximates the expectation only. As we mentioned in the previous sections, overestimation is attributed to the error in function approximation. Following Chua et al. (2018), we briefly review different uncertainty modeling techniques.
Probabilistic models output a distribution (e.g., mean and variance of a Gaussian distribution) instead of an expectation. We take the reward component of the environment model as an example,
the probabilistic model is written as r ∼ N (f̂r,ζr , σ̂2r,ζr ), and the loss function is the negative log likelihood(equation. 4).
L ζr
= −Eτ [ log pN (rt|f̂r,ζr (st, at, st+1), σ̂2r,ζr (st, at, st+1)) ] (4)
Ensemble of deterministic(DE) models maintains an ensemble of parameters, which is typically trained with different training samples and different initial parameters. E.g, given the ensemble of parameters ζr,1, ζr,2, ..., ζr,N , the expectation and the variance of the prediction is acquired from equation. 6
Êζr [ f̂r,ζr ] = 1
N ∑ i f̂r,ζr,i , V̂ζr [ f̂r,ζr ] = 1 N ∑ i (f̂r,ζr,i − Êζr [ f̂r,ζr ] )2 (5)
We define the operator Êx and V̂x as the average and the variance on the ensemble of xes respectively. As proposed by (Chua et al., 2018), the variance σ̂2 in equation. 4 mainly captures the aleatoric uncertainty, and the variance V̂ mainly captures the epistemic uncertainty.
Ensemble of probabilistic models(PE) keeps track of an collection of distributions {N (f̂r,ζr,i , σ̂2r,ζr,i)}, i ∈ [1, N ], which can further give the estimation of both uncertainties. A sampling form PE goes as follow
Sample i unformly from [1, N ], then sample r ∼ N (f̂r,ζr,i , σ̂2r,ζr,i) (6)

heading: 3.4 MODEL-BASED VALUE EXPANSION
text: In MVE(Feinberg et al., 2018) the learned environment model f̂ζr,ζs,ζd = (f̂s,ζs , f̂r,ζr , f̂d,ζd) together with the policy µθ are used to image a trajectory starting from state st and action at, which is represented by τ̂ζr,ζs,ζd,θ′,H(rt, st+1). It produces a trajectory up to horizon H(H ≥ 0). We can write τ̂ζr,ζs,ζd,θ′,H(rt, st+1) = (rt, st+1, ât+1, r̂t+1, ŝt+2, ..., ŝt+H+1, ât+H+1).
Then the target value Q̂target in equation. 2 is replaced with estimated return Q̂MVEζr,ζs,ζd,θ′,φ′,H on the sampled trajectory τ̂ζr,ζs,ζd,θ′,H(rt, st+1), which is expressed in equation 7.
Q̂target(rt, st+1)← Q̂MVEζr,ζs,ζd,θ′,φ′,H(rt, st+1)
= rt + t+H∑ t′=t+1 γt ′−tdt,t′ r̂t′ + γ H+1dt,t+H+1Q̂φ′(ŝt+H+1, ât+H+1)
with dt,t′ = (1− d(st+1)) t′∏
k=t+2
(1− f̂d,ζd(ŝk))
(7)

heading: 3.5 STOCHASTIC ENSEMBLE VALUE EXPANSION
text: Selecting proper horizon H for value expansion is important to achieve high sample efficiency and asymptotic accuracy at the same time. Though the increase of H brings increasing prophecy, the asymptotic accuracy is sacrificed due to the increasing reliance on the environment model. In STEVE(Buckman et al., 2018), interpolating the value expansions Q̂MVEζr,ζs,ζd,θ′,φ′,H of different H ∈ [0, Hmax] is proposed. The weight for each horizon is decided by the inverse of its variance, which is calculated by estimating an ensemble of values switching the combination of parameters in environment model f̂ and value function Q̂φ′ . Through our notation, STEVE can be written as equation. 8.
Q̂STEVE(rt, st+1) =
∑Hmax H=0 ωH Êζr,ζs,ζd,φ′ [ Q̂MVEζr,ζs,ζd,θ′,φ′,H(rt, st+1) ] ∑Hmax H=0 ωH ,
with ωH = V̂ζr,ζs,ζd,φ′ [ Q̂MVEζr,ζs,ζd,θ′,φ′,H(rt, st+1) ]−1 (8)

heading: 4 INVESTIGATION OF THE APPROXIMATION ERROR IN STOCHASTIC ENVIRONMENTS
text: To thoroughly investigate the impact of aleatoric uncertainty on hybri-RL methods, we construct a demonstrative toy environment(fig. 1(a)). The agent starts from s0 = 0, chooses an action at from A = [−1,+1] at each time step t. The transition of the environment is st+1 = st + at|at| + k · N (0, 1). We compare two different environments:k = 0 and k = 1, where k = 0 represents the deterministic transition, and k = 1 represents the stochastic transition. The episode terminates at (|s| > 5 ), where the agent acquires a final reward. The agent gets a constant penalty(-100) at each time step to encourage it to reach the terminal state as soon as possible. Note that the deterministic environment actually requires more steps in expectation to reach |s > 5| compared with the stochastic environment, thus the value function at the starting point of k = 1 (Ground truth = 380+) tends to be lower than that of k = 0(Ground truth = 430+).
We apply the learning methods including DDPG, MVE, STEVE to this environment, and plot the changes of estimate values at the starting point(see fig. 5).
The results show that, in the deterministic environment, the Q-values estimated by all methods converge to the ground-truth asymptotically in such a simple environment. However, after adding the noise, previous MFRL and Hybrid-RL methods show various level of overestimation. The authors of (Feinberg et al., 2018) have claimed that value expansion improves the quality of estimated values, but MVE and STEVE actually give even worse prediction than model-free DDPG in the stochastic environment. A potential explanation is that the overall overestimation comes from the unavoidable imprecision of the estimator(Thrun & Schwartz, 1993; Fujimoto et al., 2018), but Hybrid-RL also suffers from the approximation error of the dynamics model. When using a deterministic environment model, the predictive transition of both environments would be identical, because the deterministic dynamics model tends to approximate the expectation of next states(e.g, f̂s,ζs(st = 0, at > 0) = 1.0, f̂s,ζs(st = 1.0, at > 0) = 2.0). This would result in the same value estimation for k = 0 and k = 1 for both value expansion methods, but the ground truth of Q-values are different in these two environments. As a result, the deterministic environment introduces additional approximation error, leading to more severe overestimation.

heading: 5 METHODOLOGY

heading: 5.1 RISK AVERSE VALUE EXPANSION
text: We proposed mainly two improvements based on MVE and STEVE. Firstly, we apply an ensemble of probabilistic models (PE) to enable the environment model to capture the uncertainty over possible trajectories. Secondly, inspired by risk sensitive RL, we calculate the confidence lower bound of the target value(fig 1(c)).
Before introducing RAVE we start with the Distributional Value Expansion(DVE). Compared with MVE that uses a deterministic environment model and value function, DVE uses a probabilistic environment model, and we independently sample the reward and the next state using the probabilistic environment models(see equation 9 and fig. 2).
s̃t+2 ∼ N (f̂s,ζs(st+1, ãt+1 = µθ′(st+1)), σ̂2s,ζs(st+1, ãt+1))
r̃t+1 ∼ N (f̂r,ζr (st+1, ãt+1, s̃t+2), σ̂2r,ζr (st+1, ãt+1, s̃t+2))
d̃(s̃t+2) ∼ N (f̂d,ζd(s̃t+2), σ̂2d,ζd(s̃t+2))
(9)
We apply the distributional expansion starting from rt, st+1 to acquire the trajectory τ̃ζr,ζs,ζd,θ′,H(rt, st+1) = (rt, st+1, ãt+1, r̃t+1, s̃t+2, ..., s̃t+H+1, ãt+H+1), based on which we write DVE as equation. 10.
Q̂DVEζr,ζs,ζd,θ′,φ′,H(rt, st+1)
= rt + t+H∑ t′=t+1 γt ′−tdt,t′ r̃t′ + γ H+1dt,t+H+1Q̂φ′(s̃t+H+1, ãt+H+1)
with dt,t′ = (1− d(st+1)) t′∏
k=t+2
(1− d̃(s̃k))
(10)
We then keep track of an ensemble of the combination of the parameters {ζr, ζs, ζd, φ′}. For each group of parameters we use an ensemble of N parameters, which gives us 4N parameters in all. Then we select a random combination of four integers {i, j, k, l}, which gives the parameter combination of {ζr,i, ζs,j , ζd,k, φ′l}. By switching the combination of integers we acquire an ensemble of DVE estimation. Then we count the average and the variance on the ensemble of DVE, and by subtracting a certain proportion(α) of the standard variance, we acquire a lower bound of DVE estimation. We call this estimation of value function the α-confidence lower bound(α-CLB), written as equation. 11.
Q̂α−CLBH (rt, st+1) = Êζr,ζs,ζd,φ′ [ Q̂DVEζr,ζs,ζd,θ′,φ′,H(rt, st+1) ] − α √ V̂ζr,ζs,ζd,φ′ [ Q̂DVEζr,ζs,ζd,θ′,φ′,H(rt, st+1)
] (11) Subtraction of variances is commonly used in risk-sensitive RL(Sato & Kobayashi, 2000; Pan et al., 2019; Reddy et al., 2019). The motivation is straight forward, we try to suppress the utility of the
high-variance trajectories, in order to avoid possible risks. However, the α here is left undecided. We will come to this problem in the next part.
Finally, we define RAVE, which adopts the similar interpolation among different horizons as STEVE based on DVE and CLB, shown in equation. 13.
Q̂target(rt, st+1)← Q̂RAVE(rt, st+1) = ∑Hmax H=0 ωHQ̂ α−CLB H (rt, st+1)∑Hmax
H=0 ωH ,
with ωH = V̂ζr,ζs,ζd,φ′ [ Q̂DVEζr,ζs,ζd,θ′,φ′,H(rt, st+1) ]−1 (12) While adopting the lower confidence bound may introduce the bias of underestimation, it makes the policy less preferable to actions with large variance of future return.

heading: 5.2 ADAPTIVE CONFIDENCE BOUND
text: An unsolved problem in RAVE is to select proper α. The requirement of risk aversion and exploration is somehow competing: risk aversion seek to minimize the variance, while exploration searches states with higher variance. To satisfy both requirements, previous work proposed two competing agents, and each will make decision for a short amount of time(Pan et al. (2019)). Here we propose another solution to this problem. We argue that the agent needs to aggressively explore at the beginning, and it should get more risk sensitive as the model converges. A key indicator of this is the epistemic uncertainty. The epistemic uncertainty measures how well our model get to know the state space. In MBRL and Hybrid-RL, there is a common technique to easily monitor the epistemic uncertainty, by evaluating the ability of the learned environment model to predict the consequence of its own actions(Pathak et al., 2017).
Following this motivation, we set the confidence bound factor to be related to its current state and action, denoted as α(s, a). We want α(s, a) to be larger when the environment model could perfectly predict the state to get more risk sensitive, and smaller when the prediction is noisy to allow more exploration. We have
α(st, at) = max{0, α(1.0− 1
Z ||Eζs [f̂s,ζs(st, at)]− st+1||2)}, (13)
where Z is a scaling factor for the prediction error. With a little abuse of notations, we use α here to represent a constant hyperparameter, and α(s, a) is the factor that is actually used in α-CLB. α(st, at) picks the value near zero at first, and gradually increases to α with the learning process.

heading: 6 EXPERIMENTS AND ANALYSIS
text: We evaluate RAVE on continuous control environments using the MuJoCo physics simulator(Todorov et al., 2012). The baselines includes the model-free DDPG and STEVE that currently yields the SOTA Hybrid-RL performance in MuJoCo. We also align our performance with the SOTA MFRL methods including twin delayed deep deterministic (TD3) policy gradient algorithm(Fujimoto et al., 2018), soft actror-critic(SAC) algorithm(Haarnoja et al., 2018), and proximal policy optimization(PPO)(Schulman et al., 2017), using the implementation provided by the authors. To further demonstrate the robustness in complex environments, we also evaluate RAVE on OpenAI’s Roboschool (Klimov & Schulman, 2017), where STEVE has shown a large improvement than the other baselines.We detail hyper-parameters and the implementation in the supplementary materials.

heading: 6.1 EXPERIMENTAL RESULTS
text: We carried out experiments on eight environments shown in fig. 3. Among the compared methods, PPO is the only on-policy updating method, which has very poor sample-efficiency compared with either STEVE or off-policy MFRL methods, as PPO needs a large batch size to learn stably(Haarnoja et al., 2018). DDPG achieves quite high performance in HalfCheetah-v1 and Swimmer-v1, but fails on almost all the other environments, especially on challenging environments such as Humanoid.
In Hopper-v1 and Walker2d-v1, STEVE can not compare with TD3 and SAC, which yields quite good performance in many environments. However, RAVE performed favorably in most environments in both asymptotic performance and the rising speed(meaning sample efficiency), except for HalfCheetah-v1 and Swimmer-v1, where DDPG has already achieved satisfying performance and the margin between DDPG and Hybrid-RL is not that large.

heading: 6.2 ANALYSIS
text: Distribution of Value Function Approximation. To investigate whether the proposed method predicts value function more precisely, we plot the of the predicted values Q̂ against the ground truth values of Hopper-v1 in fig. 4. The ground truth value here is calculated by directly adding the rewards of the left trajectory, thus it is more like a monte carlo sampling from ground truth distribution, which is quite noisy. To better demonstrate the distribution of points, we draw the confidence ellipses representing the density. The points are extracted from the model at environment steps of 1M. In DDPG and STEVE, the predicted value and ground truth aligned poorly with the ground truth, while RAVE yields better alignments, though a little bit of underestimation.
.
Investigation on dynamic confidence bound. In order to study the role played by the α-confidence lower bound separately, we further carried out series of ablative experiments in Hopper-v1 environment. We compare RAVE(α =constant), RAVE(dynamic α(s, a)) and other baselines in chart. For all the algorithms, we set Hmax = 3, and the experiments are replicated for 4 times.
From fig. 5(a) we can see that α = 0(which means ensemble of DVE only) already surpasses the performance of STEVE in Hopper-v1, showing that modeling aleatoric uncertainty through PE indeed benefits the performance of value expansion. Larger margin is attained by introducing αCLB. A very large α(such as constant α = 2.0, which means lower CLB) can quickly stabilize the performance, but its performance stayed low due to lack of exploration, while a smaller α(constant α = 0.0, 0.5 generates larger fluctuation in performance. The dynamic adjustment of α facilitates quick rise and stable performance.
Analysis on Robustness. We also investigate the robustness of RAVE and baselines on the most challenging Mujoco environment, Humanoid-v1. Humanoid-v1 involve complex humanoid dynamics, where the agent is prone to fall. We evaluate the robustness with the possibility of falling by the learned policy. As shown in fig.5(b), RAVE achieves the lowest falling rate compared with the baselines.
Computational Complexity. A main concern toward RAVE may be its computational complexity. On the one hand, hybrid-RL involves the usage of environment model which introduces additional computational cost. On the other hand, RAVE and STEVE involves ensemble of trajectory rollouts, which is a little bit costly. We keep the ensemble size the same as STEVE and the details about the hyper-parameter can be found in the supplements.
For the training stages, the additional training cost of RAVE compared with STEVE comes from modeling aleatoric uncertainty and additional sampling cost. We tested the training speed of STEVE and RAVE, and the time for RAVE to finish training 500 batches with a batch size 512 is 13.20s, an increase of 24.29%, compared to STEVE(10.62s). The time reported here is tested in 2 P40 Nvidia GPUs with 8 CPUs(2.00GHz). For the inference stages, RAVE charges exactly the same computational resources just as the other model-free actor critic methods as long as the model architecture of the policy is equal, which is a lot more cost efficient compared with MBRL that adopts a planning procedure.
Also we want to emphasize here that the computation complexity is typically less important compared with sample efficiency, as the interaction cost matters more than computational cost in training procedure.

heading: 7 CONCLUSION
text: In this paper, we raise the problem of incomplete modeling of uncertainty and insufficient robustness in model-based value expansion. We introduce ensemble of probabilistic models to approximate the environment, based on which we introduce the distributional value expansion(DVE), α-Confidence Lower Bound, which further leads to RAVE. Our experiments demonstrate the superiority of RAVE in both sample efficiency and robustness, compared with state-of-the-art RL methods, including the model-free TD3 algorithm and the Hybrid-RL STEVE algorithm. We hope that this algorithm will facilitate the application of reinforcement learning in real-world, complex and risky scenarios.

heading: A TRAINING AND IMPLEMENTATION DETAILS
text: A.1 NEURAL NETWORK STRUCTURE
We used rectified linear units (ReLUs) between all hidden layers of all our implemented algorithms. Unless otherwise stated, all the output layers of model have no activation function. RL Models. We implement model-based algorithms on top of DDPG, with a policy network and a Q-value network. The policy network is a stack of 4 fully-connected(FC) layers. The activation function of the output layer is tanh to constrain the output range of the network. The Q-value network takes the concatenation of the state st and the action at as input, followed by four FC layers.
Dynamics Models. We train three neural networks as the transition function, the reward function and the termination function. We build eight FC layers for the transition approximator, and four FC layers for the other approximators. The distributional models N (f̂ , σ̂2) in RAVE use the similar model structure except that there are two output layers corresponding to the mean and the variances respectively.
A.2 PARALLEL TRAINING
We use distributed training to accelerate our algorithms and the baseline algorithms. Following the implementation of STEVE, we train a GPU learner with multiple actors deployed in a CPU cluster. The actors reload the parameters periodically from the learner, generate trajectories and send the trajectories to the learner. The learner stores them in the replay buffer, and updates the parameters with data randomly sampled from the buffer. For the network communication, we use Ray(Moritz et al., 2018) to transfer data and parameters between the actors and the learner. We have 8 actors generating data, and deploy the learner on the GPUs. DDPG uses a GPU, and model-based methods uses two: one for the training of the policy and another for the dynamics model.
A.3 ROLLOUT DETAILS
We employ the identical method of target candidates computation as STEVE, except we image a rollout with an ensemble of probabilistic models . At first, we bind the parameters of transition model(ζs,i) to the termination model(ζd,i). That is, we numerate the combination of three integers {i, j, k|i, j, k ∈ [1, N ]}, which gives us an ensemble of N3 parameters {ζr,j , ζs,i, ζd,i φ′k}. The actual sampling process goes like this: For each H ∈ [0, Hmax], we first use the transition model(ζs,i) and the termination model(ζd,i) to image a state-action sequence {st+1, ãt+1, s̃t+2, ãt+2, ..., s̃t+H+1, ãt+H+1}; Based on the state-action sequence, we use reward function(ζr,j) to estimate the rewards (r̂t′ = f̂r,ζr,j (s̃t′ , ãt′ , s̃t′+1)) and the value function(φ ′ k) to predict the value of the last state(Q̂φ′k(s̃t+H+1, ãt+H+1))(fig. 6). In total we predict N 3(Hmax+1) combination of rewards and value functions in both RAVE and STEVE.

heading: B HYPER-PARAMETERS FOR TRAINING
text: We list all the hyper-parameters used in our experiments in table 1.

Review summaries:
This paper proposes a novel deep reinforcement learning algorithm at the intersection of model-based and model-free reinforcement learning: Risk Averse Value Expansion (RAVE). Overall, this work represents a significant but incremental step forwards for this "hybrid"-RL class of algorithms. However, the paper itself has significant weaknesses in its writing, analysis, and presentation of ideas.

This paper expands on previous work on hybrid model-based and model-free reinforcement learning. Specifically, it expands on the ideas in Model-based Value Expansion (MVE) and Stochastic Ensemble Value Expansion (STEVE) with a dynamically-scaled variance bias term to increase risk aversion over the course of learning, which the authors call Risk Averse Value Expansion (RAVE). Experimental results indicate notable improvements over their selected model-free and hybrid RL baselines on continuous control tasks in terms of initial learning efficiency (how many environment steps are needed to achieve a particular level of performance), asymptotic performance (how high the performance is given the same large number of environment steps), and avoidance of negative outcomes (how infrequently major negative outcomes are encountered over the course of training).