Title: RISK AVERSE VALUE EXPANSION FOR SAMPLE EFFI-CIENT AND ROBUST POLICY LEARNING

Abstract: Model-based Reinforcement Learning(RL) has shown great advantage in sampleefficiency, but suffers from poor asymptotic performance and high inference cost. A promising direction is to combine model-based reinforcement learning with model-free reinforcement learning, such as model-based value expansion(MVE). However, the previous methods do not take into account the stochastic character of the environment, thus still suffers from higher function approximation errors. As a result, they tend to fall behind the best model-free algorithms in some challenging scenarios. We propose a novel Hybrid-RL method, which is developed from MVE, namely the Risk Averse Value Expansion(RAVE). In the proposed method, we use an ensemble of probabilistic models for environment modeling to generate imaginative rollouts, based on which we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on different environments including MuJoCo and robo-school show that RAVE yields state-of-the-art performance. Also we found that it greatly prevented some catastrophic consequences such as falling down and thus reduced the variance of the rewards.

1. INTRODUCTION:

In contrast to the tremendous progress made by model-free reinforcement learning algorithms in the domain of games (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019) , poor sample efficiency has risen up as a great challenge to RL, especially when interacting with the real world. Toward this challenge, a promising direction is to integrate the dynamics model to enhance the sample efficiency of the learning process (Sutton, 1991; Calandra et al., 2016; Kalweit & Boedecker, 2017; Oh et al., 2017; Racanière et al., 2017) . However, classic model-based reinforcement learning(MBRL) methods tend to lag behind the model-free methods(MFRL) asymptotically, especially in cases of noisy environments and long trajectories. The hybrid combination of MFRL and MBRL(Hybrid-RL for short) has attracted much attention due to this reason. A lot of efforts has been devoted to this field, including the dyna algorithm (Kurutach et al., 2018) , model-based value expansion (Feinberg et al., 2018) , I2A (Weber et al., 2017) , etc.

The robustness of the learned policy is another concern in RL. For stochastic environments, policy can be vulnerable to tiny disturbance and occasionally drop into catastrophic consequences. In MFRL, off-policy RL(such as DQN, DDPG) typically suffers from such problems, which in the end leads to instability in the performance including sudden drop in the rewards. To solve such problem, risk sensitive MFRL not only maximize the expected return, but also try to reduce those catastrophic outcomes (Garcıa & Fernández, 2015; Dabney et al., 2018a; Pan et al., 2019) . For MBRL and Hybrid-RL, without modeling the uncertainty in the environment(especially for continuous states and actions), it often leads to higher function approximation errors and poorer performances. It is proposed that complete modeling of uncertainty in transition can obviously improve the performance (Chua et al., 2018) , however, reducing risks in MBRL and Hybrid-RL has not been sufficiently studied yet.

In order to achieve both sample efficiency and robustness at the same time, we propose a new Hybrid-RL method more capable of solving stochastic and risky environments. The proposed method, namely Risk Averse Value Expansion(RAVE), is an extension of the model-based value expansion(MVE) (Feinberg et al., 2018) and stochastic ensemble value expansion(STEVE) (Buckman et al., 2018) . We systematically analyse the approximation errors of different methods in stochastic environments. We borrow ideas from the uncertainty modeling ( Chua et al. (2018) ) and risk averse reinforcement learning. The probabilistic ensemble environment model is used, which captures not only the variance in estimation(also called epistemic uncertainty), but also stochastic transition nature of the environment(also called aleatoric uncertainty). Utilizing the ensemble of estimations, we further adopt a dynamic confidence lower bound of the target value function to make the policy more risk-sensitive. We compare RAVE with prior MFRL and Hybrid-RL baselines, showing that RAVE not only yields SOTA expected performance, but also facilitates the robustness of the policy.

2. RELATED WORKS:

The model-based value expansion(MVE) (Feinberg et al., 2018 ) is a Hybrid-RL algorithm. Unlike typical MFRL such as DQN that uses only 1 step bootstrapping, MVE uses the imagination rollouts of length H to predict the target value. The assistance of environment model can greatly improve the sample efficiency at the start, but the precision of long term inference becomes limited asymptotically. In order to properly balance the contribution of the value expansion of different horizons, stochastic ensemble value expansion(STEVE) (Buckman et al., 2018) adopts an interpolation of value expansion of different horizon. The accuracy of the expansion is estimated through the ensemble of environment models as well as value functions. Ensemble of environment models also models the uncertainty to some extent, however, ensemble of deterministic model captures mainly epistemic uncertainty instead of stochastic transitions (Chua et al., 2018) .

The uncertainty or the function approximation error is typically divided into three classes (Geman et al., 1992) : the noise exists in the objective environment, e.g., the stochastic transitions, which is also called aleatoric uncertainty (Chua et al., 2018) . The model bias is the error produced by the limited expressive power of the approximating function, which is measured by the expectation of ground truth and the prediction of the model, in case that infinite training data is provided. The variance is the uncertainty brought by insufficient training data, which is also called epistemic uncertainty. Dabney et al. (2018b) discuss the epistemic and aleatoric uncertainty in their work and focus on the latter one to improve the distributional RL. Recent work suggests that ensemble of probabilistic model(PE) is considered as more thorough modeling of uncertainty (Chua et al., 2018) , while simply aggregate deterministic model captures only variance or epistemic uncertainty. The stochastic transition is more related to the noise(or aleatoric uncertainty), and the epistemic uncertainty is usually of interest to many works in terms of exploitation&exploration (Pathak et al., 2017; Schmidhuber, 2010; Oudeyer & Kaplan, 2009) . Other works adopt ensemble of deterministic value function for exploration (Osband et al., 2016; Buckman et al., 2018) .

Risks in RL typically refer to the inherent uncertainty of the environment and the fact that policy may perform poorly in some cases (Garcıa & Fernández, 2015) . Risk sensitive learning requires not only maximization of expected rewards, but also lower variances and risks in performance. Toward this object, some works adopt the variance of the return (Sato et al., 2001; Pan et al., 2019; Reddy et al., 2019) , or the worst-case outcome (Heger, 1994; Gaskett, 2003) in either policy learning (Pan et al., 2019; Reddy et al., 2019) , exploration (Smirnova et al., 2019) , or distributional value estimates (Dabney et al., 2018a ). An interesting issue in risk reduction is that reduction of risks is typically found to be conflicting with exploration and exploitation that try to maximize the reward in the long run. Authors in (Pan et al., 2019) introduce two adversarial agents(risk aversion and longterm reward seeking) that act in combination to solve this problem. Still, it remains quite tricky and empiristic to trade-off between risk-sensitive and risk-seeking(exploration) in RL. In this paper, we propose a dynamic confidence bound for this purpose.

A number of prior works have studied function approximation error that leads to overestimation and sub-optimal solution in MFRL. Double DQN(DDQN) (Van Hasselt et al., 2016) improves over DQN through disentangling the target value function and the target policy that pursues maximum value. In TD3 (Fujimoto et al., 2018) the authors suggest that systematic overestimation of value function also exists in actor-critic MFRL. They use an ensemble of two value functions, with the minimum estimate being used as the target value. Selecting the lower value estimation is similar to using uncertainty or lower confidence bound which is adopted by the other risk sensitive methods (Pan et al., 2019) , though they claimed different motivations.

3. PRELIMINARIES:



3.1. ACTOR-CRITIC MODEL-FREE REINFORCEMENT LEARNING:

The Markov Decision Processes(MDP) is used to describe the process of an agent interacting with the environment. The agent selects the action a t ∈ A at each time step t. After executing the action, it receives a new observation s t+1 ∈ S and a feedback r t ∈ R from the environment. As we focus mainly on the environments of continuous action, we denote the parametric deterministic policy that the agent uses to decide its action as a t = µ θ (s t ). Typically we add Gaussian exploration noise on top of the deterministic policy, such that we have a stochastic behavioral policy π θ,σ : S ×A → R. It is calculated as π θ,σ (s t , a t ) = p N (a t |µ θ (s t ), σ 2 ), where p N (x|m, σ 2 ) represents the probability density at x in a Gaussian distribution N (m, σ 2 ). As the interaction process continues, the agent generates a trajectory τ = (s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ...) following the policy π θ,σ . For finite horizon MDP, we use the indicator d : S → {0, 1} to mark whether the episode is terminated. The objective of RL is to find the optimal policy π * to maximize the expected discounted sum of rewards along the trajectory. The value performing the action a with policy π at the state s is defined by

Q π (s, a) = E s0=s,a0=a,τ ∼π ∞ t=0 γ t r t ,

where 0 < γ < 1 is the discount factor. The value iteration in model-free RL tries to approximate the optimal value Q π * with a parametric value function Qφ by minimizing the Temporal Difference(TD) error, where φ is the parameter to be optimized. The TD error between the estimates of Q-value and the corresponding target values is shown in equation. 1, where φ is a delayed copy of the parameter φ, and a ∼ π θ , with θ being a delayed copy of θ (Lillicrap et al., 2015) .

EQUATION

To optimize the deterministic policy function in a continuous action space, deep deterministic policy gradient(DDPG) (Lillicrap et al., 2015) maximizes the value function (or minimizes the negative value function) under the policy µ θ with respect to parameter θ, shown in equation. 2.

L θ = -E τ [ t Qφ (s t , µ θ (s t ))]

(2)

3.2. ENVIRONMENT MODELING:

To model the environment in continuous space, an environment model is typically composed of three individual mapping functions: fr,ζr : 

S × A × S → R,

and this process might go on to generate a complete imagined trajectory of [s t , a t , rt , ŝt+1 , ...].

The neural network is commonly used as an environment model due to its powerful express ability.

To optimize the parameter ζ we need to minimize the mean square error(or the cross entropy) of the prediction and the ground truth, given the trajectories τ under the behavioral policy.

3.3. UNCERTAINTY AWARE PREDICTION:

The deterministic model approximates the expectation only. As we mentioned in the previous sections, overestimation is attributed to the error in function approximation. Following Chua et al. (2018) , we briefly review different uncertainty modeling techniques.

Probabilistic models output a distribution (e.g., mean and variance of a Gaussian distribution) instead of an expectation. We take the reward component of the environment model as an example, the probabilistic model is written as r ∼ N ( fr,ζr , σ2 r,ζr ), and the loss function is the negative log likelihood(equation. 4). 

L ζr = -E τ log p N (

We define the operator Êx and Vx as the average and the variance on the ensemble of xes respectively. As proposed by (Chua et al., 2018) , the variance σ2 in equation. 4 mainly captures the aleatoric uncertainty, and the variance V mainly captures the epistemic uncertainty. 

EQUATION

3.5. STOCHASTIC ENSEMBLE VALUE EXPANSION:

Selecting proper horizon H for value expansion is important to achieve high sample efficiency and asymptotic accuracy at the same time. Though the increase of H brings increasing prophecy, the asymptotic accuracy is sacrificed due to the increasing reliance on the environment model.

In STEVE (Buckman et al., 2018) , interpolating the value expansions QMVE ζr,ζs,ζ d ,θ ,φ ,H of different H ∈ [0, H max ] is proposed. The weight for each horizon is decided by the inverse of its variance, which is calculated by estimating an ensemble of values switching the combination of parameters in environment model f and value function Qφ . Through our notation, STEVE can be written as equation. 8. We apply the learning methods including DDPG, MVE, STEVE to this environment, and plot the changes of estimate values at the starting point(see fig. 5 ).

QSTEVE (r:

The results show that, in the deterministic environment, the Q-values estimated by all methods converge to the ground-truth asymptotically in such a simple environment. However, after adding the noise, previous MFRL and Hybrid-RL methods show various level of overestimation. The authors of (Feinberg et al., 2018) have claimed that value expansion improves the quality of estimated values, but MVE and STEVE actually give even worse prediction than model-free DDPG in the stochastic environment. A potential explanation is that the overall overestimation comes from the unavoidable imprecision of the estimator (Thrun & Schwartz, 1993; Fujimoto et al., 2018) , but Hybrid-RL also suffers from the approximation error of the dynamics model. When using a deterministic environment model, the predictive transition of both environments would be identical, because the deterministic dynamics model tends to approximate the expectation of next states(e.g, fs,ζs (s t = 0, a t > 0) = 1.0, fs,ζs (s t = 1.0, a t > 0) = 2.0). This would result in the same value estimation for k = 0 and k = 1 for both value expansion methods, but the ground truth of Q-values are different in these two environments. As a result, the deterministic environment introduces additional approximation error, leading to more severe overestimation. 

We apply the distributional expansion starting from r t , s t+1 to acquire the trajectory τζr,ζs,ζ d ,θ ,H (r t , s t+1 ) = (r t , s t+1 , ãt+1 , rt+1 , st+2 , ..., st+H+1 , ãt+H+1 ), based on which we write DVE as equation. 10.

EQUATION

We then keep track of an ensemble of the combination of the parameters {ζ r , ζ s , ζ d , φ }. For each group of parameters we use an ensemble of N parameters, which gives us 4N parameters in all.

Then we select a random combination of four integers {i, j, k, l}, which gives the parameter combination of {ζ r,i , ζ s,j , ζ d,k , φ l }. By switching the combination of integers we acquire an ensemble of DVE estimation. Then we count the average and the variance on the ensemble of DVE, and by subtracting a certain proportion(α) of the standard variance, we acquire a lower bound of DVE estimation. We call this estimation of value function the α-confidence lower bound(α-CLB), written as equation. 11. 

Qα:

Subtraction of variances is commonly used in risk-sensitive RL (Sato & Kobayashi, 2000; Pan et al., 2019; Reddy et al., 2019) . The motivation is straight forward, we try to suppress the utility of the high-variance trajectories, in order to avoid possible risks. However, the α here is left undecided. We will come to this problem in the next part. 

While adopting the lower confidence bound may introduce the bias of underestimation, it makes the policy less preferable to actions with large variance of return.

5.2. ADAPTIVE CONFIDENCE BOUND:

An unsolved problem in RAVE is to select proper α. The requirement of risk aversion and exploration is somehow competing: risk aversion seek to minimize the variance, while exploration searches states with higher variance. To satisfy both requirements, previous work proposed two competing agents, and each will make decision for a short amount of time (Pan et al. (2019) ). Here we propose another solution to this problem. We argue that the agent needs to aggressively explore at the beginning, and it should get more risk sensitive as the model converges. A key indicator of this is the epistemic uncertainty. The epistemic uncertainty measures how well our model get to know the state space. In MBRL and Hybrid-RL, there is a common technique to easily monitor the epistemic uncertainty, by evaluating the ability of the learned environment model to predict the consequence of its own actions (Pathak et al., 2017) .

Following this motivation, we set the confidence bound factor to be related to its current state and action, denoted as α(s, a). We want α(s, a) to be larger when the environment model could perfectly predict the state to get more risk sensitive, and smaller when the prediction is noisy to allow more exploration. We have

EQUATION

)

where Z is a scaling factor for the prediction error. With a little abuse of notations, we use α here to represent a constant hyperparameter, and α(s, a) is the factor that is actually used in α-CLB. α(s t , a t ) picks the value near zero at first, and gradually increases to α with the learning process.

6. EXPERIMENTS AND ANALYSIS:

We evaluate RAVE on continuous control environments using the MuJoCo physics simulator (Todorov et al., 2012) . The baselines includes the model-free DDPG and STEVE that currently yields the SOTA Hybrid-RL performance in MuJoCo. We also align our performance with the SOTA MFRL methods including twin delayed deep deterministic (TD3) policy gradient algorithm (Fujimoto et al., 2018) , soft actror-critic(SAC) algorithm (Haarnoja et al., 2018) , and proximal policy optimization(PPO) (Schulman et al., 2017) , using the implementation provided by the authors. To further demonstrate the robustness in complex environments, we also evaluate RAVE on OpenAI's Roboschool (Klimov & Schulman, 2017) , where STEVE has shown a large improvement than the other baselines.We detail hyper-parameters and the implementation in the supplementary materials.

6.1. EXPERIMENTAL RESULTS:

We carried out experiments on eight environments shown in fig. 3 . Among the compared methods, PPO is the only on-policy updating method, which has very poor sample-efficiency compared with either STEVE or off-policy MFRL methods, as PPO needs a large batch size to learn stably (Haarnoja et al., 2018) . DDPG achieves quite high performance in HalfCheetah-v1 and Swimmer-v1, but fails on almost all the other environments, especially on challenging environments such as Humanoid. In Hopper-v1 and Walker2d-v1, STEVE can not compare with TD3 and SAC, which yields quite good performance in many environments. However, RAVE performed favorably in most environments in both asymptotic performance and the rising speed(meaning sample efficiency), except for HalfCheetah-v1 and Swimmer-v1, where DDPG has already achieved satisfying performance and the margin between DDPG and Hybrid-RL is not that large.

6.2. ANALYSIS:

Distribution of Value Function Approximation. To investigate whether the proposed method predicts value function more precisely, we plot the of the predicted values Q against the ground truth values of Hopper-v1 in fig. 4 . The ground truth value here is calculated by directly adding the rewards of the left trajectory, thus it is more like a monte carlo sampling from ground truth distribution, which is quite noisy. To better demonstrate the distribution of points, we draw the confidence ellipses representing the density. The points are extracted from the model at environment steps of 1M. In DDPG and STEVE, the predicted value and ground truth aligned poorly with the ground truth, while RAVE yields better alignments, though a little bit of underestimation. 

:



Figures and Tables:

INVESTIGATION OF THE APPROXIMATION ERROR IN STOCHASTICENVIRONMENTSTo thoroughly investigate the impact of aleatoric uncertainty on hybri-RL methods, we construct a demonstrative toy environment(fig.1(a)). The agent starts from s 0 = 0, chooses an action a t from A = [-1, +1] at each time step t. The transition of the environment is s t+1 = s t + at |at| + k • N (0, 1). We compare two different environments:k = 0 and k = 1, where k = 0 represents the deterministic transition, and k = 1 represents the stochastic transition. The episode terminates at (|s| > 5 ), where the agent acquires a final reward. The agent gets a constant penalty(-100) at each time step to encourage it to reach the terminal state as soon as possible. Note that the deterministic environment actually requires more steps in expectation to reach |s > 5| compared with the stochastic environment, thus the value function at the starting point of k = 1 (Ground truth = 380+) tends to be lower than that of k = 0(Ground truth = 430+).

Figure 1: Curves of estimated Q values in the toy environment at the starting point over the environment steps.Each experiment is run four times.

two improvements based on MVE and STEVE. Firstly, we apply an ensemble of probabilistic models (PE) to enable the environment model to capture the uncertainty over possible trajectories. Secondly, inspired by risk sensitive RL, we calculate the confidence lower bound of the target value(fig 1(c)).

Figure 2: A casual illustration of three Distributional Value Expansion(DVE) trajectories. Before introducing RAVE we start with the Distributional Value Expansion(DVE). Compared with MVE that uses a deterministic environment model and value function, DVE uses a probabilistic environment model, and we independently sample the reward and the next state using the probabilistic environment models(see equation 9 and fig. 2). st+2 ∼ N ( fs,ζs (s t+1 , ãt+1 = µ θ (s t+1 )), σ2 s,ζs (s t+1 , ãt+1 )) rt+1 ∼ N ( fr,ζr (s t+1 , ãt+1 , st+2 ), σ2 r,ζr (s t+1 , ãt+1 , st+2 )) d(s t+2 ) ∼ N ( fd,ζ d (s t+2 ), σ2 d,ζ d (s t+2 ))

Figure 3: Average return over environment frames in MuJoCo and Roboschool environments. Each experiment is run four times.

Figure 4: Confidence ellipses of the distribution of estimated values Q versus the ground truth in Hopper-v1. The points are extracted from environment steps of 1M, each includes statistics from 10,000 points. The x-axis and y-axis represent the statistical cumulative discounted returns(ground truth) and the predictive Q-values(both are normalized), respectively..

r t | fr,ζr (s t , a t , s t+1 ), σ2 r,ζr (s t , a t , s t+1 )) (4) Ensemble of deterministic(DE) models maintains an ensemble of parameters, which is typically trained with different training samples and different initial parameters. E.g, given the ensemble of parameters ζ r,1 , ζ r,2 , ..., ζ r,N , the expectation and the variance of the prediction is acquired from equation. 6

Table content: <table><tr><td>Êζr fr,ζr =</td><td>1 N</td><td>i</td><td>fr,ζr,i , Vζr fr,ζr =</td><td>1 N</td></tr></table>

Ensemble of probabilistic models(PE) keeps track of an collection of distributions {N ( fr,ζr,i , σ2 r,ζr,i )}, i ∈ [1, N ], which can further give the estimation of both uncertainties. A sampling form PE goes as follow Sample i unformly from[1, N ], then sample r ∼ N ( fr,ζr,i , σ2 on the sampled trajectory τζr,ζs,ζ d ,θ ,H (r t , s t+1 ), which is expressed in equation 7.

Table content: <table><tr><td>Qtarget (r</td></tr></table>

, s t+1 ) = Êζr,ζs,ζ d ,φ QDVE ζr,ζs,ζ d ,θ ,φ ,H (r t , s t+1 ) -α Vζr,ζs,ζ d ,φ QDVE ζr,ζs,ζ d ,θ ,φ ,H (r t , s t+1 )

Table content: <table/>

Finally, we define RAVE, which adopts the similar interpolation among different horizons as STEVE based on DVE and CLB, shown in equation. 13. Qtarget (r t , s t+1 ) ← QRAVE (r t , s t+1 ) =

Table content: <table><tr><td/><td>Hmax H=0 ω H</td><td colspan="2">Qα-CLB H Hmax H=0 ω H (r t , s t+1 )</td><td>,</td></tr><tr><td>with ω H = Vζr,ζs,ζ d ,φ</td><td colspan="2">ζr,ζs,ζ d ,θ ,φ ,H (r t , s t+1 ) QDVE</td><td>-1</td></tr></table>

Peer review (iclr2020):
This paper considers the problem of high function approximation errors facing the stochastic environments when trying to combining model-based reinforcement learning (RL) with model-free RL. The paper begins by showing that previous methods like model-based value expansion (MVE) and stochastic ensemble value expansion (STEVE) can perform even worse than the pure model-free DDPG algorithm in a rather noisy environment. Following the ideas of MVE and STEVE, It then proposes a risk averse value expansion (RAVE) to replace the target Q function in the actor-critic algorithm, which is built upon an ensemble of probabilistic models (PE) and adopt the lower confidence bound as a surrogate of the target value as in the risk-sensitive RL. A simple yet intuitive approach for adaptively selecting the confidence bound alpha is also proposed. The experiments show that RAVE does improve over the state-of-the-art algorithms in several different environments, with a better draw-down control. In general, this paper is well-written and the idea of RAVE is novel as far as I know. But since I*m not very familiar with the specific literature of combing model-based and model-free RL, and since the idea of RAVE is relatively straightforward (but admittedly practically powerful and theoretically interesting), I choose to give a conservative accept to account the possibility that some existing works have followed very similar approaches. Some minor comments: 1) What is the loss of the negative log-likelihood in (4) for? 2) The authors may want to explain clearly what is the variance indicating in (8) more clearly, although one can guess that it is closely related to (5).