Understanding the Asymptotic Performance of Model-Based RL Methods

1 INTRODUCTION . Model-based reinforcement learning ( MBRL ) has many potential benefits over model-free approaches . These include ( i ) the ability to generalize to new tasks in the environment , without having to retrain ; ( ii ) learning from off-policy data and ( iii ) sample efficiency . However , in simulated environments where data is plentiful , model-based approaches struggle to approach the asymptotic performance of model-free methods Nagabandi et al . ( 2017 ) ; Pong et al . ( 2018 ) ; Chua et al . ( 2018 ) . Several possible explanations present themselves : the planner used for selecting optimal actions under the model might be insufficiently powerful ; the model might not be able to accurately model the dynamics ; or the planning horizon might not be long enough . This paper address these questions by teasing apart the different factors involved in an MBRL framework , applied to two deterministic MuJoCo environments ( Todorov et al. , 2012 ) , with the aim of understanding the gap in asymptotic performance with respect to model-free approaches . In particular , we demonstrate that bias caused by short planning horizons and poor accuracy of long-term predictions is the cause of the poor performance of existing MBRL methods in the unlimited-sample regime . Our experiments show that , with a perfect dynamics model , the optimal planing horizon can be over 100 steps – much longer than typically considered in many MBRL approaches . Correspondingly , the performance is typically limited by the ability of the model to accurately predict over long-time scales , not just a few time-steps . Existing approaches to MBRL rely on a single-step dynamics model that predict the next state , given the current state and an action . As can be see in Figure 5 , over long time-horizons the errors compound due to recursive application of the model , yielding inaccurate state estimates which are not useful for planning . Instead , we propose an alternate form of dynamics model that takes as input a sequence of actions along with the current state and directly predicts many time-steps into the future . This approach provides accurate prediction over long time horizons , allowing us to uncover the relationship between model accuracy and performance . This reveals that MBRL with sufficiently good learned models matches or exceeds the performance of state-of-the-art model-free methods . 1.1 RELATED WORK . Non-Parametric Model-Based RL : Gaussian processes are popular approach to modeling nonlinear dynamics due to their low sample complexity and their ability to explicitly represent epistemic uncertainty . Consequently , numerous MBRL approaches use them , e.g . Kocijan et al . ( 2004 ) ; Ko et al . ( 2007 ) ; Grancharova et al . ( 2008 ) ; Deisenroth & Rasmussen ( 2011 ) ; Deisenroth et al . ( 2014 ) . However via the choice of kernel they impose potentially unrealistic smoothness constraints and do not scale to large data settings , limiting their asymptotic performance in practice . Combining model-based and model-free methods : Due to the sample efficiency of model-based methods and superior asymptotic performance of model-free methods , several works have proposed to learn dynamics models using a few trajectory samples , then use those models to train or augment a model-free policy . The classic Dyna algorithm ( Sutton , 1990 ) uses a model to extend Bellman updates multiple steps . Deisenroth & Rasmussen ( 2011 ) learns a Gaussian process model of the dynamics function and uses it to train an RBF network policy , and Gal et al . ( 2016 ) enables the model to scale to larger data by using Bayesian neural networks in place of GPs . Levine et al . ( 2016 ) fits a time-varying locally linear model around a trajectory , then trains a neural network policy to follow trajectories found by iLQR ( Todorov & Li , 2005 ) . Silver et al . ( 2016 ) learns an implicit model of the dynamics for implicit planning via value estimation ; in an inversion of this technique , Pong et al . ( 2018 ) learn an explicit model of Q values for explicit planning via constrained optimization . Weber et al . ( 2017 ) learns a neural network dynamics model which is unrolled inside a policy to inform an actor-critic agent . Nagabandi et al . ( 2017 ) trains a neural network dynamics model on control tasks and uses it to take actions , then uses that model-based policy to speed the training of a model-free policy via imitation learning . These works largely seek to either ( i ) augment a modelfree method with a model for faster learning , or ( ii ) make up for the asymptotic deficiencies of a model-based method by transitioning to model-free . In this work we instead directly investigate the causes of MBRL ’ s poor asymptotic performance with the aim of making a transition to model-free unnecessary . MBRL with neural network models : The idea of using neural networks to enable model-based control of nonlinear systems goes back decades ( Miller et al. , 1990 ; Schmidhuber , 1990 ; Hunt et al. , 1992 ; Bekey & Goldberg , 2012 ; Draeger et al. , 1995 ) , but until recently has only seen significant success on systems with relatively simple dynamics . Several works have endeavored to use neural network generative models of images for model-based control ( Wahlström et al. , 2015 ; Watter et al. , 2015 ; Finn & Levine , 2017 ) ; these policies have typically used short planning horizons and struggled to equal model-free performance on complex tasks . Lenz et al . ( 2015 ) learn recurrent neural network dynamics models , then use backpropagation through time to select actions and control a robotic arm , and Henaff et al . ( 2017 ) extend this concept to both discrete and continuous action spaces . Clavera et al . ( 2018 ) combine meta-learning with MBRL using neural network models to rapidly adapt to novel environments . Srinivas et al . ( 2018 ) uses imitation learning to train a model to plan by gradient descent , which relies on an existing expert rather than learning from scratch . The closest work to ours is Chua et al . ( 2018 ) . This follows a similar recipe , with similar planning and constructing a dataset online , but with different models and different goals . We use deterministic neural networks which predict many steps into the future to understand the impact of model- and horizon-bias on asymptotic performance of MBRL methods on long-horizon problems . Chua et al . ( 2018 ) uses a bootstrapped ensemble of probabilistic neural networks to improve the performance of MBRL in the few-sample regime . While that work achieves strong performance on short-horizon tasks , in our experiments we find it struggles to equal model-free methods on tasks with very long horizons . 2 APPROACH . In this section we describe the models used in our experiments , the action-conditional predictor ( ACP ) and the novel plan-conditional predictor ( PCP ) , which predicts the outcome of a sequence of actions with a single model step . We then detail the framework we use for planning with and training these models . 2.1 NOTATION . We denote states and actions at a time t by st and at . In the environments we consider , both st and at are continuous vectors . We use H to refer to the planning horizon of an MPC policy . We refer to a sequence of actions as a plan ; a plan constructed with horizon H is thus p = { a1 , ... , aH } . In a set of plans { p1 , ... , pn } , aij refers to the jth action of the ith plan . We consider models which predict a future state given the current state and one or more actions . We denote R to be the range of such a model , which is the number of steps this model predicts in a single application : fR ( st , at , . . . , at+R−1 ; θ ) = s̃t+R , for parameters θ . We apply the model recursively using the notation FT ( st , at , . . . , aT−1 ; θ ) = fR ( . . . fR ( fR ( st , at , . . . , at+R−1 ; θ ) , at+R , . . . , at+2R−1 ; θ ) . . . , aT−R , . . . , aT−1 ; θ ) = s̃T . That is , FT ( ) applies fR ( ) recursively T/R times . 2.2 PLAN-CONDITIONAL PREDICTORS . To test our conjecture that compounding errors limit the asymptotic performance of existing models on long-horizon RL tasks , we propose plan-conditional predictors ( PCPs ) . A PCP takes the form fR ( st , at , . . . , at+R−1 ) = s̃t+R for some range R > 1 . If R = 1 , then it reduces to the standard approach ( Deisenroth & Rasmussen , 2011 ; Gal et al. , 2016 ; Henaff et al. , 2017 ; Nagabandi et al. , 2017 ; Chua et al. , 2018 ) which predicts only a single time-step at a time , and which we call an action-conditional predictor ( ACP ) . As shown in Figure 1 , a PCP can predict H steps into the future using H/R recursive applications of the model instead of H applications required by an actionconditional predictor . There are many possible parameterizations that could be used in a PCP . In this work we choose deep fully-connected neural networks . There are several reasons for this : ( i ) the space of inputs grows exponentially with the range R , thus models with high capacity are needed to minimize model bias ; ( ii ) since the goal is to understand asymptotic performance , a large data regime is assumed and sample complexity is a secondary issue ; ( iii ) by predicting R steps with a single network application , they are extremely fast at planning time . An obvious alternate parameterization is recurrent neural networks ( RNNs ) , and we provide experimental comparisons between the two approaches in Section 3 . For both Swimmer and HalfCheetah , the ACP and PCP networks consist of fully-connected networks with 9 hidden layers with 1000 units each using the SELU activation function ( Klambauer et al. , 2017 ) . The input state and action ( s ) are concatenated before being used as input to the network . The same loss function is used for training PCPs and ACPs , namely ‖s̃t+R − st+R‖22 , where s is the raw MuJoCo state . We assume that the environment forms a Markov decision process ( MDP ) ( Bellman , 1957 ) with deterministic dynamics , properties shared by the tasks considered in this work . These assumptions allow us to focus exclusively on the significance of model fidelity and planning horizon to MBRL , but removing them is an interesting direction for future work . 2.2.1 INTERMEDIATE PREDICTIONS . For visualization purposes , in some experiments we use a variant of a plan-conditional predictor which takes as input a state and a variable number of actions R′ where 1 ≤ R′ ≤ R , the remainder of action inputs being set to zero . This allows us to plot error or render video of the PCP ’ s predictions at each intermediate timestep instead of only at multiples of R. Furthermore , this variant of the model allows for planning based on reward functions which operate at each timestep rather than just at the end of the episode ( see Section 2.3.1 ) . As such , this variant of the model is applicable to any environment .

Summary: The authors learn a model that predicts the state R steps in the future, given the current state and intervening actions, instead of the predicting the next time step state. The model is then used for standard model predictive control. The authors find numerically that their method, termed Plan-Conditional Predictor (PCP), performs better over long horizon times (~100 time steps), than other recent model-based and model-free algorithms. This because for long horizon time scales, the model predicting the state for the next time step accumulates error when used recursively.