Risk Averse Value Expansion for Sample Efficient and Robust Policy Learning

1 INTRODUCTION . In contrast to the tremendous progress made by model-free reinforcement learning algorithms in the domain of games ( Mnih et al. , 2015 ; Silver et al. , 2017 ; Vinyals et al. , 2019 ) , poor sample efficiency has risen up as a great challenge to RL , especially when interacting with the real world . Toward this challenge , a promising direction is to integrate the dynamics model to enhance the sample efficiency of the learning process ( Sutton , 1991 ; Calandra et al. , 2016 ; Kalweit & Boedecker , 2017 ; Oh et al. , 2017 ; Racanière et al. , 2017 ) . However , classic model-based reinforcement learning ( MBRL ) methods tend to lag behind the model-free methods ( MFRL ) asymptotically , especially in cases of noisy environments and long trajectories . The hybrid combination of MFRL and MBRL ( Hybrid-RL for short ) has attracted much attention due to this reason . A lot of efforts has been devoted to this field , including the dyna algorithm ( Kurutach et al. , 2018 ) , model-based value expansion ( Feinberg et al. , 2018 ) , I2A ( Weber et al. , 2017 ) , etc . The robustness of the learned policy is another concern in RL . For stochastic environments , policy can be vulnerable to tiny disturbance and occasionally drop into catastrophic consequences . In MFRL , off-policy RL ( such as DQN , DDPG ) typically suffers from such problems , which in the end leads to instability in the performance including sudden drop in the rewards . To solve such problem , risk sensitive MFRL not only maximize the expected return , but also try to reduce those catastrophic outcomes ( Garcıa & Fernández , 2015 ; Dabney et al. , 2018a ; Pan et al. , 2019 ) . For MBRL and Hybrid-RL , without modeling the uncertainty in the environment ( especially for continuous states and actions ) , it often leads to higher function approximation errors and poorer performances . It is proposed that complete modeling of uncertainty in transition can obviously improve the performance ( Chua et al. , 2018 ) , however , reducing risks in MBRL and Hybrid-RL has not been sufficiently studied yet . In order to achieve both sample efficiency and robustness at the same time , we propose a new Hybrid-RL method more capable of solving stochastic and risky environments . The proposed method , namely Risk Averse Value Expansion ( RAVE ) , is an extension of the model-based value expansion ( MVE ) ( Feinberg et al. , 2018 ) and stochastic ensemble value expansion ( STEVE ) ( Buckman et al. , 2018 ) . We systematically analyse the approximation errors of different methods in stochastic environments . We borrow ideas from the uncertainty modeling ( Chua et al . ( 2018 ) ) and risk averse reinforcement learning . The probabilistic ensemble environment model is used , which captures not only the variance in estimation ( also called epistemic uncertainty ) , but also stochastic transition nature of the environment ( also called aleatoric uncertainty ) . Utilizing the ensemble of estimations , we further adopt a dynamic confidence lower bound of the target value function to make the policy more risk-sensitive . We compare RAVE with prior MFRL and Hybrid-RL baselines , showing that RAVE not only yields SOTA expected performance , but also facilitates the robustness of the policy . 2 RELATED WORKS . The model-based value expansion ( MVE ) ( Feinberg et al. , 2018 ) is a Hybrid-RL algorithm . Unlike typical MFRL such as DQN that uses only 1 step bootstrapping , MVE uses the imagination rollouts of length H to predict the target value . The assistance of environment model can greatly improve the sample efficiency at the start , but the precision of long term inference becomes limited asymptotically . In order to properly balance the contribution of the value expansion of different horizons , stochastic ensemble value expansion ( STEVE ) ( Buckman et al. , 2018 ) adopts an interpolation of value expansion of different horizon . The accuracy of the expansion is estimated through the ensemble of environment models as well as value functions . Ensemble of environment models also models the uncertainty to some extent , however , ensemble of deterministic model captures mainly epistemic uncertainty instead of stochastic transitions ( Chua et al. , 2018 ) . The uncertainty or the function approximation error is typically divided into three classes ( Geman et al. , 1992 ) : the noise exists in the objective environment , e.g. , the stochastic transitions , which is also called aleatoric uncertainty ( Chua et al. , 2018 ) . The model bias is the error produced by the limited expressive power of the approximating function , which is measured by the expectation of ground truth and the prediction of the model , in case that infinite training data is provided . The variance is the uncertainty brought by insufficient training data , which is also called epistemic uncertainty . Dabney et al . ( 2018b ) discuss the epistemic and aleatoric uncertainty in their work and focus on the latter one to improve the distributional RL . Recent work suggests that ensemble of probabilistic model ( PE ) is considered as more thorough modeling of uncertainty ( Chua et al. , 2018 ) , while simply aggregate deterministic model captures only variance or epistemic uncertainty . The stochastic transition is more related to the noise ( or aleatoric uncertainty ) , and the epistemic uncertainty is usually of interest to many works in terms of exploitation & exploration ( Pathak et al. , 2017 ; Schmidhuber , 2010 ; Oudeyer & Kaplan , 2009 ) . Other works adopt ensemble of deterministic value function for exploration ( Osband et al. , 2016 ; Buckman et al. , 2018 ) . Risks in RL typically refer to the inherent uncertainty of the environment and the fact that policy may perform poorly in some cases ( Garcıa & Fernández , 2015 ) . Risk sensitive learning requires not only maximization of expected rewards , but also lower variances and risks in performance . Toward this object , some works adopt the variance of the return ( Sato et al. , 2001 ; Pan et al. , 2019 ; Reddy et al. , 2019 ) , or the worst-case outcome ( Heger , 1994 ; Gaskett , 2003 ) in either policy learning ( Pan et al. , 2019 ; Reddy et al. , 2019 ) , exploration ( Smirnova et al. , 2019 ) , or distributional value estimates ( Dabney et al. , 2018a ) . An interesting issue in risk reduction is that reduction of risks is typically found to be conflicting with exploration and exploitation that try to maximize the reward in the long run . Authors in ( Pan et al. , 2019 ) introduce two adversarial agents ( risk aversion and longterm reward seeking ) that act in combination to solve this problem . Still , it remains quite tricky and empiristic to trade-off between risk-sensitive and risk-seeking ( exploration ) in RL . In this paper , we propose a dynamic confidence bound for this purpose . A number of prior works have studied function approximation error that leads to overestimation and sub-optimal solution in MFRL . Double DQN ( DDQN ) ( Van Hasselt et al. , 2016 ) improves over DQN through disentangling the target value function and the target policy that pursues maximum value . In TD3 ( Fujimoto et al. , 2018 ) the authors suggest that systematic overestimation of value function also exists in actor-critic MFRL . They use an ensemble of two value functions , with the minimum estimate being used as the target value . Selecting the lower value estimation is similar to using uncertainty or lower confidence bound which is adopted by the other risk sensitive methods ( Pan et al. , 2019 ) , though they claimed different motivations . 3 PRELIMINARIES . 3.1 ACTOR-CRITIC MODEL-FREE REINFORCEMENT LEARNING . The Markov Decision Processes ( MDP ) is used to describe the process of an agent interacting with the environment . The agent selects the action at ∈ A at each time step t. After executing the action , it receives a new observation st+1 ∈ S and a feedback rt ∈ R from the environment . As we focus mainly on the environments of continuous action , we denote the parametric deterministic policy that the agent uses to decide its action as at = µθ ( st ) . Typically we add Gaussian exploration noise on top of the deterministic policy , such that we have a stochastic behavioral policy πθ , σ : S×A → R. It is calculated as πθ , σ ( st , at ) = pN ( at|µθ ( st ) , σ2 ) , where pN ( x|m , σ2 ) represents the probability density at x in a Gaussian distribution N ( m , σ2 ) . As the interaction process continues , the agent generates a trajectory τ = ( s0 , a0 , r0 , s1 , a1 , r1 , ... ) following the policy πθ , σ . For finite horizon MDP , we use the indicator d : S → { 0 , 1 } to mark whether the episode is terminated . The objective of RL is to find the optimal policy π∗ to maximize the expected discounted sum of rewards along the trajectory . The value performing the action a with policy π at the state s is defined by Qπ ( s , a ) = Es0=s , a0=a , τ∼π ∑∞ t=0 γ trt , where 0 < γ < 1 is the discount factor . The value iteration in model-free RL tries to approximate the optimal valueQπ ∗ with a parametric value function Q̂φ by minimizing the Temporal Difference ( TD ) error , where φ is the parameter to be optimized . The TD error between the estimates of Q-value and the corresponding target values is shown in equation . 1 , where φ′ is a delayed copy of the parameter φ , and a′ ∼ πθ′ , with θ′ being a delayed copy of θ ( Lillicrap et al. , 2015 ) . L φ = Eτ [ ∑ t ( Q̂target ( rt , st+1 ) − Q̂φ ( st , at ) ) 2 ] with Q̂target ( rt , st+1 ) = rt + γ · ( 1− d ( st+1 ) ) · Q̂φ′ ( st+1 , a′ ) ) ( 1 ) To optimize the deterministic policy function in a continuous action space , deep deterministic policy gradient ( DDPG ) ( Lillicrap et al. , 2015 ) maximizes the value function ( or minimizes the negative value function ) under the policy µθ with respect to parameter θ , shown in equation . 2 . L θ = −Eτ [ ∑ t Q̂φ′ ( st , µθ ( st ) ) ] ( 2 ) 3.2 ENVIRONMENT MODELING . To model the environment in continuous space , an environment model is typically composed of three individual mapping functions : f̂r , ζr : S × A× S → R , f̂s , ζs : S × A → S , and f̂d , ζd : S → [ 0 , 1 ] , which are used to approximate the feedback , next state and probability of the terminal indicator respectively ( Gu et al. , 2016 ; Feinberg et al. , 2018 ) . Here ζr , ζs and ζd are used to represent the parameters of the corresponding mapping functions . With the environment model , starting from st , at , we can predict the next state and reward by ŝt+1 = f̂s , ζs ( st , at ) , r̂t = f̂r , ζr ( st , at , ŝt+1 ) , d̂t+1 = f̂d , ζd ( ŝt+1 ) , ( 3 ) and this process might go on to generate a complete imagined trajectory of [ st , at , r̂t , ŝt+1 , ... ] . The neural network is commonly used as an environment model due to its powerful express ability . To optimize the parameter ζ we need to minimize the mean square error ( or the cross entropy ) of the prediction and the ground truth , given the trajectories τ under the behavioral policy .

Summary: This paper expands on previous work on hybrid model-based and model-free reinforcement learning. Specifically, it expands on the ideas in Model-based Value Expansion (MVE) and Stochastic Ensemble Value Expansion (STEVE) with a dynamically-scaled variance bias term to increase risk aversion over the course of learning, which the authors call Risk Averse Value Expansion (RAVE). Experimental results indicate notable improvements over their selected model-free and hybrid RL baselines on continuous control tasks in terms of initial learning efficiency (how many environment steps are needed to achieve a particular level of performance), asymptotic performance (how high the performance is given the same large number of environment steps), and avoidance of negative outcomes (how infrequently major negative outcomes are encountered over the course of training).