Risk Averse Value Expansion for Sample Efficient and Robust Policy Learning

1 INTRODUCTION . In contrast to the tremendous progress made by model-free reinforcement learning algorithms in the domain of games ( Mnih et al. , 2015 ; Silver et al. , 2017 ; Vinyals et al. , 2019 ) , poor sample efficiency has risen up as a great challenge to RL , especially when interacting with the real world . Toward this challenge , a promising direction is to integrate the dynamics model to enhance the sample efficiency of the learning process ( Sutton , 1991 ; Calandra et al. , 2016 ; Kalweit & Boedecker , 2017 ; Oh et al. , 2017 ; Racanière et al. , 2017 ) . However , classic model-based reinforcement learning ( MBRL ) methods tend to lag behind the model-free methods ( MFRL ) asymptotically , especially in cases of noisy environments and long trajectories . The hybrid combination of MFRL and MBRL ( Hybrid-RL for short ) has attracted much attention due to this reason . A lot of efforts has been devoted to this field , including the dyna algorithm ( Kurutach et al. , 2018 ) , model-based value expansion ( Feinberg et al. , 2018 ) , I2A ( Weber et al. , 2017 ) , etc . The robustness of the learned policy is another concern in RL . For stochastic environments , policy can be vulnerable to tiny disturbance and occasionally drop into catastrophic consequences . In MFRL , off-policy RL ( such as DQN , DDPG ) typically suffers from such problems , which in the end leads to instability in the performance including sudden drop in the rewards . To solve such problem , risk sensitive MFRL not only maximize the expected return , but also try to reduce those catastrophic outcomes ( Garcıa & Fernández , 2015 ; Dabney et al. , 2018a ; Pan et al. , 2019 ) . For MBRL and Hybrid-RL , without modeling the uncertainty in the environment ( especially for continuous states and actions ) , it often leads to higher function approximation errors and poorer performances . It is proposed that complete modeling of uncertainty in transition can obviously improve the performance ( Chua et al. , 2018 ) , however , reducing risks in MBRL and Hybrid-RL has not been sufficiently studied yet . In order to achieve both sample efficiency and robustness at the same time , we propose a new Hybrid-RL method more capable of solving stochastic and risky environments . The proposed method , namely Risk Averse Value Expansion ( RAVE ) , is an extension of the model-based value expansion ( MVE ) ( Feinberg et al. , 2018 ) and stochastic ensemble value expansion ( STEVE ) ( Buckman et al. , 2018 ) . We systematically analyse the approximation errors of different methods in stochastic environments . We borrow ideas from the uncertainty modeling ( Chua et al . ( 2018 ) ) and risk averse reinforcement learning . The probabilistic ensemble environment model is used , which captures not only the variance in estimation ( also called epistemic uncertainty ) , but also stochastic transition nature of the environment ( also called aleatoric uncertainty ) . Utilizing the ensemble of estimations , we further adopt a dynamic confidence lower bound of the target value function to make the policy more risk-sensitive . We compare RAVE with prior MFRL and Hybrid-RL baselines , showing that RAVE not only yields SOTA expected performance , but also facilitates the robustness of the policy . 2 RELATED WORKS . The model-based value expansion ( MVE ) ( Feinberg et al. , 2018 ) is a Hybrid-RL algorithm . Unlike typical MFRL such as DQN that uses only 1 step bootstrapping , MVE uses the imagination rollouts of length H to predict the target value . The assistance of environment model can greatly improve the sample efficiency at the start , but the precision of long term inference becomes limited asymptotically . In order to properly balance the contribution of the value expansion of different horizons , stochastic ensemble value expansion ( STEVE ) ( Buckman et al. , 2018 ) adopts an interpolation of value expansion of different horizon . The accuracy of the expansion is estimated through the ensemble of environment models as well as value functions . Ensemble of environment models also models the uncertainty to some extent , however , ensemble of deterministic model captures mainly epistemic uncertainty instead of stochastic transitions ( Chua et al. , 2018 ) . The uncertainty or the function approximation error is typically divided into three classes ( Geman et al. , 1992 ) : the noise exists in the objective environment , e.g. , the stochastic transitions , which is also called aleatoric uncertainty ( Chua et al. , 2018 ) . The model bias is the error produced by the limited expressive power of the approximating function , which is measured by the expectation of ground truth and the prediction of the model , in case that infinite training data is provided . The variance is the uncertainty brought by insufficient training data , which is also called epistemic uncertainty . Dabney et al . ( 2018b ) discuss the epistemic and aleatoric uncertainty in their work and focus on the latter one to improve the distributional RL . Recent work suggests that ensemble of probabilistic model ( PE ) is considered as more thorough modeling of uncertainty ( Chua et al. , 2018 ) , while simply aggregate deterministic model captures only variance or epistemic uncertainty . The stochastic transition is more related to the noise ( or aleatoric uncertainty ) , and the epistemic uncertainty is usually of interest to many works in terms of exploitation & exploration ( Pathak et al. , 2017 ; Schmidhuber , 2010 ; Oudeyer & Kaplan , 2009 ) . Other works adopt ensemble of deterministic value function for exploration ( Osband et al. , 2016 ; Buckman et al. , 2018 ) . Risks in RL typically refer to the inherent uncertainty of the environment and the fact that policy may perform poorly in some cases ( Garcıa & Fernández , 2015 ) . Risk sensitive learning requires not only maximization of expected rewards , but also lower variances and risks in performance . Toward this object , some works adopt the variance of the return ( Sato et al. , 2001 ; Pan et al. , 2019 ; Reddy et al. , 2019 ) , or the worst-case outcome ( Heger , 1994 ; Gaskett , 2003 ) in either policy learning ( Pan et al. , 2019 ; Reddy et al. , 2019 ) , exploration ( Smirnova et al. , 2019 ) , or distributional value estimates ( Dabney et al. , 2018a ) . An interesting issue in risk reduction is that reduction of risks is typically found to be conflicting with exploration and exploitation that try to maximize the reward in the long run . Authors in ( Pan et al. , 2019 ) introduce two adversarial agents ( risk aversion and longterm reward seeking ) that act in combination to solve this problem . Still , it remains quite tricky and empiristic to trade-off between risk-sensitive and risk-seeking ( exploration ) in RL . In this paper , we propose a dynamic confidence bound for this purpose . A number of prior works have studied function approximation error that leads to overestimation and sub-optimal solution in MFRL . Double DQN ( DDQN ) ( Van Hasselt et al. , 2016 ) improves over DQN through disentangling the target value function and the target policy that pursues maximum value . In TD3 ( Fujimoto et al. , 2018 ) the authors suggest that systematic overestimation of value function also exists in actor-critic MFRL . They use an ensemble of two value functions , with the minimum estimate being used as the target value . Selecting the lower value estimation is similar to using uncertainty or lower confidence bound which is adopted by the other risk sensitive methods ( Pan et al. , 2019 ) , though they claimed different motivations . 3 PRELIMINARIES . 3.1 ACTOR-CRITIC MODEL-FREE REINFORCEMENT LEARNING . The Markov Decision Processes ( MDP ) is used to describe the process of an agent interacting with the environment . The agent selects the action at ∈ A at each time step t. After executing the action , it receives a new observation st+1 ∈ S and a feedback rt ∈ R from the environment . As we focus mainly on the environments of continuous action , we denote the parametric deterministic policy that the agent uses to decide its action as at = µθ ( st ) . Typically we add Gaussian exploration noise on top of the deterministic policy , such that we have a stochastic behavioral policy πθ , σ : S×A → R. It is calculated as πθ , σ ( st , at ) = pN ( at|µθ ( st ) , σ2 ) , where pN ( x|m , σ2 ) represents the probability density at x in a Gaussian distribution N ( m , σ2 ) . As the interaction process continues , the agent generates a trajectory τ = ( s0 , a0 , r0 , s1 , a1 , r1 , ... ) following the policy πθ , σ . For finite horizon MDP , we use the indicator d : S → { 0 , 1 } to mark whether the episode is terminated . The objective of RL is to find the optimal policy π∗ to maximize the expected discounted sum of rewards along the trajectory . The value performing the action a with policy π at the state s is defined by Qπ ( s , a ) = Es0=s , a0=a , τ∼π ∑∞ t=0 γ trt , where 0 < γ < 1 is the discount factor . The value iteration in model-free RL tries to approximate the optimal valueQπ ∗ with a parametric value function Q̂φ by minimizing the Temporal Difference ( TD ) error , where φ is the parameter to be optimized . The TD error between the estimates of Q-value and the corresponding target values is shown in equation . 1 , where φ′ is a delayed copy of the parameter φ , and a′ ∼ πθ′ , with θ′ being a delayed copy of θ ( Lillicrap et al. , 2015 ) . L φ = Eτ [ ∑ t ( Q̂target ( rt , st+1 ) − Q̂φ ( st , at ) ) 2 ] with Q̂target ( rt , st+1 ) = rt + γ · ( 1− d ( st+1 ) ) · Q̂φ′ ( st+1 , a′ ) ) ( 1 ) To optimize the deterministic policy function in a continuous action space , deep deterministic policy gradient ( DDPG ) ( Lillicrap et al. , 2015 ) maximizes the value function ( or minimizes the negative value function ) under the policy µθ with respect to parameter θ , shown in equation . 2 . L θ = −Eτ [ ∑ t Q̂φ′ ( st , µθ ( st ) ) ] ( 2 ) 3.2 ENVIRONMENT MODELING . To model the environment in continuous space , an environment model is typically composed of three individual mapping functions : f̂r , ζr : S × A× S → R , f̂s , ζs : S × A → S , and f̂d , ζd : S → [ 0 , 1 ] , which are used to approximate the feedback , next state and probability of the terminal indicator respectively ( Gu et al. , 2016 ; Feinberg et al. , 2018 ) . Here ζr , ζs and ζd are used to represent the parameters of the corresponding mapping functions . With the environment model , starting from st , at , we can predict the next state and reward by ŝt+1 = f̂s , ζs ( st , at ) , r̂t = f̂r , ζr ( st , at , ŝt+1 ) , d̂t+1 = f̂d , ζd ( ŝt+1 ) , ( 3 ) and this process might go on to generate a complete imagined trajectory of [ st , at , r̂t , ŝt+1 , ... ] . The neural network is commonly used as an environment model due to its powerful express ability . To optimize the parameter ζ we need to minimize the mean square error ( or the cross entropy ) of the prediction and the ground truth , given the trajectories τ under the behavioral policy .

Summary: This paper proposes a novel deep reinforcement learning algorithm at the intersection of model-based and model-free reinforcement learning: Risk Averse Value Expansion (RAVE). Overall, this work represents a significant but incremental step forwards for this "hybrid"-RL class of algorithms. However, the paper itself has significant weaknesses in its writing, analysis, and presentation of ideas.