Policy Optimization In the Face of Uncertainty

Model-based reinforcement learning has the potential to be more sample efficient than model-free approaches . However , existing model-based methods are vulnerable to model bias , which leads to poor generalization and asymptotic performance compared to model-free counterparts . In this paper , we propose a novel policy optimization framework using an uncertainty-aware objective function to handle those issues . In this framework , the agent simultaneously learns an uncertaintyaware dynamics model and optimizes the policy according to these learned models . Under this framework , the objective function can represented end-to-end as a single computational graph , which allows seamless policy gradient computation via backpropagation through the models . In addition to being theoretically sound , our approach shows promising results on challenging continuous control benchmarks with competitive asymptotic performance and sample complexity compared to state-of-the-art baselines . 1 INTRODUCTION . Popular reinforcement learning ( RL ) algorithms are divided into two main paradigms : model-free ( MFRL ) and model-based ( MBRL ) types . While achieving good asymtotic performances in many high dimensional problems ( Mnih et al. , 2015 ; Silver et al. , 2017 ; Schulman et al. , 2017 ; Hessel et al. , 2018 ; Espeholt et al. , 2018 ) , MFRL methods suffer from high sample complexity since they learn state/state-action values only from rewards and do not explicitly exploit the rich information underlying the transition dynamics data . On the contrary , MBRL approaches , by trying model the transition dynamics that are in turn used for planning without having to frequently interacting with real systems , are known to have sample efficiency and thus possess more practicability ( Deisenroth et al. , 2013 ; Finn et al. , 2016 ; Ebert et al. , 2018 ; Sutton & Barto , 2018 ; Kaiser et al. , 2019 ) . Current MBRL methods , however , still have limitations because the accuracy of the learned dynamics model is usually not satisfied , especially in complex environments ( Zhang et al. , 2018 ; Lowrey et al. , 2018 ) . The model error and its compounding effect when planning , i.e . a small bias in the model can lead to a highly erroneous value function estimate and a strongly-biased suboptimal policy , make MBRL less competitive in terms of asymptotic performance than MFRL for many non-trivial tasks . Numerous attempts have been made to tackle with this model bias problem but none of them have been really successful , such as using Gaussian Process ( GP ) ( Deisenroth & Rasmussen , 2011 ; Gal & Ghahramani , 2016 ) , Bayesian Neural Networks ( Gal et al. , 2016 ; Depeweg et al. , 2016a ; Kamthe & Deisenroth , 2017 ) , and Emsembling ( Kurutach et al. , 2018 ; Clavera et al. , 2018 ) . Another limitation of many existing MBRL methods is that they rely on the model predictive control ( MPC ) framework ( Garcia et al. , 1989 ) . While being commonly used , MPC has serveral drawbacks ( Atkeson & Schaal , 1997 ; Thananjeyan et al. , 2019 ) . First , each step requires solving a highdimensional optimization problem and thus is computationally prohibitive for applications requiring either real-time or low-latency reaction such as autonomous driving . Second , the policy is only implicit via solving the mentioned optimization problem . Not being able to explicitly represent the policy makes it hard to transfer the learned policy to other tasks or to initialize agents with an existing better-than-random policy . Contributions . To address those challenges of MBRL , we propose a new framework called Policy Optimization with Uncertainty-aware Model ( POUM ) that is able to optimize in the face of uncertainty . Our policy optimization is based on Policy Gradient , which has been widely adopted in MFRL ( Lillicrap et al. , 2015 ; Schulman et al. , 2017 ; Haarnoja et al. , 2018 ) . However , in POUM , the objective function , a utility function , is formulated around the uncertainty-aware dynamics model . This utility function takes into account both the mean and the variance of the value function estimate . This helps reducing the model bias while effectively approximating true objective , which is the value function of the policy . For experiments , we demonstrate the advantages of POUM over state-of-the-art ( SoTA ) methods on various RL tasks given training from scratch and all the environments are unaltered , and also investigate on how much risk is tolerable in those tasks . And last , POUM can be represented end-to-end in a single computation graph , which greatly facilitates the training . 2 RELATED WORK . Traditional MBRL . Initial successes of MBRL in continuous control achieved promising results by learning control policies trained on models of local dynamics using linear parametric approximators ( Abbeel et al. , 2007 ; Levine & Koltun , 2013 ) . Alternative methods such as Deisenroth & Rasmussen ( 2011 ) ; Levine & Koltun ( 2013 ) incorporated non-parametric probabilistic GPs to capture model uncertainty during policy planning and evaluation . While these methods enhance data efficiency in low-dimensional tasks , their applications in more challenging domains such as environments involving non-contact dynamics and high-dimensional control remain limited by the inflexibility of their temporally local structure and intractable inference time . Our approach , on the contrary , pushes the uncertainty modeling to the objective function and not anywhere else in the architecture . Plus , the fact that this objective is designed to propagate all the way to the value function makes it versatile in capturing uncertainty . What is more , all core components are constructed by neural networks gives our solution more power in dealing with high-dimensional tasks , thus acquiring asymptotically high performance compared to MFRL methods and , at the same time , retaining data efficiency in those complex domains . Deep Neural Networks ( DNNs ) . Recently , there has been a revived interest in using DNNs to learn predictive models of environments from data , drawing inspiration from ideas in the early literature on this MBRL field , mainly because the large representational capacity enables them as suitable function approximators for complex environments , especially that involve images or videos ( Ebert et al. , 2018 ; Kaiser et al. , 2019 ) . However , additional care has to be usually taken to avoid model bias , a situation where the DNNs overfit in the early stages of learning , resulting in inaccurate models . For example , Depeweg et al . ( 2016b ) modeled a Bayesian type of DNNs to capture uncertainty in transition dynamics . In another approach , Nagabandi et al . ( 2017 ) combined a learned dynamics network with MPC to initialize the policy network to accelerate learning in model-free deep RL . Chua et al . ( 2018 ) extended this idea by introducing a bootstrapped ensemble of probabilistic DNNs to model predictive uncertainty of the learned networks and demonstrating that a pure model-based approach can attain the asymptotic performance of MFRL counterparts . However , the use of MPC to define a policy leads to poor run-time execution and hard to transfer policy across tasks . On the contrary , our framework is much simpler in that we do not employ any extra method to model the dynamics uncertainty into DNNs that are already complicated itself with numerous architectures and hyperparameters , but instead formulate a single , new uncertainty-aware objective for end-to-end optimization . Ensemble . Another group of work leveraged the learned ensemble of dynamics models to train a policy network . Kurutach et al . ( 2018 ) learned a stochastic policy via trust-region policy optimization , and Clavera et al . ( 2018 ) casted the policy gradient as a meta-learning adaptation step with respect to each member of the ensemble . Buckman et al . ( 2018 ) proposed an algorithm to learn a weighted combination of roll-outs of different horizon lengths , which dynamically interpolates between model-based and model-free learning based on the uncertainty in the model predictions . To our knowledge , this is the closest work in aside from ours , which learns a reward function in addition to the dynamics function . But none of the aforementioned work propagates the uncertainty all the way to the value function and uses the concept of utility function to balance risk and return as in our model . Finally , ensemble of DNNs also provide a straightforward technique to obtain reliable estimates of predictive uncertainty ( Lakshminarayanan et al. , 2017 ) and has been integrated with bootstrap to guide exploration in MFRL ( Osband et al. , 2016 ; Janner et al. , 2019 ) . While many of the approaches mentioned in this section employ bootstrap to train an ensemble of models , we note that their implementations comprise of reconstructing bootstrap datasets at every training iteration , which effectively trains every single data sample and thus diminishes the advantage on uncertainty quantification achieved through bootstrapping . Except for a novel objective formulation , our model is different in that , to maintain online bootstrapped datasets across ensembles , it adds each incoming data sample to a dataset according to a Poisson probability distribution ( Park et al. , 2007 ; Qin et al. , 2013 ) , thereby guaranteeing asymptotically consistent those datasets . 3 UNCERTAINTY-AWARE MODEL-BASED POLICY OPTIMIZATION . 3.1 BACKGROUND . Consider a discrete-time Markov Decision Process ( MDP ) defined by a tuple M = { S , A , f , r , γ } , in which S is a state space , A is an action space , f : S ×A→ S is a deterministic ( or probabilistic ) transition function , r : S × A → R is a deterministic reward function , and γ ∈ ( 0 , 1 ) is a discount factor . We define the return as sum of the rewards r ( st , at ) = r ( st , π ( st ) ) for t = 0 , . . . , T for the whole trajectory ( s0 , a0 , ... , sT , aT ) induced by a policy π : S → A and discounted by γ . Here T ∈ Z+ is a task horizon , which may take a value of∞ for non-episodic environments . The goal of RL is to find an optimal policy π ? to maximize the expected return J ( π ) = E s0∼S [ V π ( s0 ) ] ( 1 ) where the value function is defined as V π ( s0 ) = ∑T−1 t=0 γ t r ( st , π ( st ) ) , and the state transition is st+1 = f ( st , π ( st ) ) , with s0 being randomly chosen from the distribution of s ∈ S. Then if the dynamics function f and the reward function r are given , solving Equation 1 can be done using the Calculus of Variations ( Young , 2000 ) or using Policy Gradient ( Sutton et al. , 2000 ) when the control function is parameterized or is finite dimensional . In RL , however , f and r are often unknown and hence Equation 1 becomes a blackbox optimization problem with an unknown objective function . Following the Bayesian approach commonly used in the blackbox optimization literature ( Shahriari et al. , 2015 ) , we propose to solve this problem by iteratively learning a probabilistic estimate V̂ of V from data and optimizing the policy according to this approximate model , as detailed in the next section . 3.2 FORMULATION OF UNCERTAINTY-AWARE OPTIMIZATION OBJECTIVE . It is worth noting that any unbiased method would model V̂ ( π ) as a probabilistic estimate , i.e . V̂ ( π ) would be a distribution ( as opposed to a point estimate ) for a given policy π. Optimizing a stochastic objective is , however , not well-defined . Our solution is to transform V̂ into a deterministic utility function that reflects a subjective measure balancing the risk and return . Following Markowitz ( 1952 ) ; Sato et al . ( 2001 ) ; Garcıa & Fernández ( 2015 ) , we propose a risk-sensitive objective criterion using a linear combination of the mean and the standard deviation of V̂ ( π ) . Formally stated , our objective criterion , which we also call the utility function , now becomes U ( π ) ( s0 ) = E s0∼S [ µ ( V̂ ( π ) ( s0 ) ) + c× σ ( V̂ ( π ) ( s0 ) ) ] , ( 2 ) where µ and σ are respectively the mean and the standard deviation of V̂ ( π ) ( s0 ) , and c is a constant that represents the subjective risk preference of the learning agent . A positive risk preference infers that the agent is adventurous while a negative risk preference indicates that the agent has a safe exploration strategy . To our best knowledge , this uncertainty-aware model-based objective function has not been used in the RL literature .

Summary: The motivation of this paper relies on the dynamics model not being accurate enough, which leads to compounding errors. Hence, a proper characterization of the uncertainty is needed. However, the model-based methods that suffer the most of this problem are the ones that are build on top of policy gradients (since they need to predict the entire trajectory) (e.g., [6]). The methods that learn a Q-value function from the model do not suffer as much from this problem since they just predict shorter horizons. Current model-based RL methods that learn a Q or value-function take into account the uncertainty (i.e., STEVE, MBPO). Those methods are not “less competitive in terms of asymptotic performance.”