Understanding the Asymptotic Performance of Model-Based RL Methods

1 INTRODUCTION . Model-based reinforcement learning ( MBRL ) has many potential benefits over model-free approaches . These include ( i ) the ability to generalize to new tasks in the environment , without having to retrain ; ( ii ) learning from off-policy data and ( iii ) sample efficiency . However , in simulated environments where data is plentiful , model-based approaches struggle to approach the asymptotic performance of model-free methods Nagabandi et al . ( 2017 ) ; Pong et al . ( 2018 ) ; Chua et al . ( 2018 ) . Several possible explanations present themselves : the planner used for selecting optimal actions under the model might be insufficiently powerful ; the model might not be able to accurately model the dynamics ; or the planning horizon might not be long enough . This paper address these questions by teasing apart the different factors involved in an MBRL framework , applied to two deterministic MuJoCo environments ( Todorov et al. , 2012 ) , with the aim of understanding the gap in asymptotic performance with respect to model-free approaches . In particular , we demonstrate that bias caused by short planning horizons and poor accuracy of long-term predictions is the cause of the poor performance of existing MBRL methods in the unlimited-sample regime . Our experiments show that , with a perfect dynamics model , the optimal planing horizon can be over 100 steps – much longer than typically considered in many MBRL approaches . Correspondingly , the performance is typically limited by the ability of the model to accurately predict over long-time scales , not just a few time-steps . Existing approaches to MBRL rely on a single-step dynamics model that predict the next state , given the current state and an action . As can be see in Figure 5 , over long time-horizons the errors compound due to recursive application of the model , yielding inaccurate state estimates which are not useful for planning . Instead , we propose an alternate form of dynamics model that takes as input a sequence of actions along with the current state and directly predicts many time-steps into the future . This approach provides accurate prediction over long time horizons , allowing us to uncover the relationship between model accuracy and performance . This reveals that MBRL with sufficiently good learned models matches or exceeds the performance of state-of-the-art model-free methods . 1.1 RELATED WORK . Non-Parametric Model-Based RL : Gaussian processes are popular approach to modeling nonlinear dynamics due to their low sample complexity and their ability to explicitly represent epistemic uncertainty . Consequently , numerous MBRL approaches use them , e.g . Kocijan et al . ( 2004 ) ; Ko et al . ( 2007 ) ; Grancharova et al . ( 2008 ) ; Deisenroth & Rasmussen ( 2011 ) ; Deisenroth et al . ( 2014 ) . However via the choice of kernel they impose potentially unrealistic smoothness constraints and do not scale to large data settings , limiting their asymptotic performance in practice . Combining model-based and model-free methods : Due to the sample efficiency of model-based methods and superior asymptotic performance of model-free methods , several works have proposed to learn dynamics models using a few trajectory samples , then use those models to train or augment a model-free policy . The classic Dyna algorithm ( Sutton , 1990 ) uses a model to extend Bellman updates multiple steps . Deisenroth & Rasmussen ( 2011 ) learns a Gaussian process model of the dynamics function and uses it to train an RBF network policy , and Gal et al . ( 2016 ) enables the model to scale to larger data by using Bayesian neural networks in place of GPs . Levine et al . ( 2016 ) fits a time-varying locally linear model around a trajectory , then trains a neural network policy to follow trajectories found by iLQR ( Todorov & Li , 2005 ) . Silver et al . ( 2016 ) learns an implicit model of the dynamics for implicit planning via value estimation ; in an inversion of this technique , Pong et al . ( 2018 ) learn an explicit model of Q values for explicit planning via constrained optimization . Weber et al . ( 2017 ) learns a neural network dynamics model which is unrolled inside a policy to inform an actor-critic agent . Nagabandi et al . ( 2017 ) trains a neural network dynamics model on control tasks and uses it to take actions , then uses that model-based policy to speed the training of a model-free policy via imitation learning . These works largely seek to either ( i ) augment a modelfree method with a model for faster learning , or ( ii ) make up for the asymptotic deficiencies of a model-based method by transitioning to model-free . In this work we instead directly investigate the causes of MBRL ’ s poor asymptotic performance with the aim of making a transition to model-free unnecessary . MBRL with neural network models : The idea of using neural networks to enable model-based control of nonlinear systems goes back decades ( Miller et al. , 1990 ; Schmidhuber , 1990 ; Hunt et al. , 1992 ; Bekey & Goldberg , 2012 ; Draeger et al. , 1995 ) , but until recently has only seen significant success on systems with relatively simple dynamics . Several works have endeavored to use neural network generative models of images for model-based control ( Wahlström et al. , 2015 ; Watter et al. , 2015 ; Finn & Levine , 2017 ) ; these policies have typically used short planning horizons and struggled to equal model-free performance on complex tasks . Lenz et al . ( 2015 ) learn recurrent neural network dynamics models , then use backpropagation through time to select actions and control a robotic arm , and Henaff et al . ( 2017 ) extend this concept to both discrete and continuous action spaces . Clavera et al . ( 2018 ) combine meta-learning with MBRL using neural network models to rapidly adapt to novel environments . Srinivas et al . ( 2018 ) uses imitation learning to train a model to plan by gradient descent , which relies on an existing expert rather than learning from scratch . The closest work to ours is Chua et al . ( 2018 ) . This follows a similar recipe , with similar planning and constructing a dataset online , but with different models and different goals . We use deterministic neural networks which predict many steps into the future to understand the impact of model- and horizon-bias on asymptotic performance of MBRL methods on long-horizon problems . Chua et al . ( 2018 ) uses a bootstrapped ensemble of probabilistic neural networks to improve the performance of MBRL in the few-sample regime . While that work achieves strong performance on short-horizon tasks , in our experiments we find it struggles to equal model-free methods on tasks with very long horizons . 2 APPROACH . In this section we describe the models used in our experiments , the action-conditional predictor ( ACP ) and the novel plan-conditional predictor ( PCP ) , which predicts the outcome of a sequence of actions with a single model step . We then detail the framework we use for planning with and training these models . 2.1 NOTATION . We denote states and actions at a time t by st and at . In the environments we consider , both st and at are continuous vectors . We use H to refer to the planning horizon of an MPC policy . We refer to a sequence of actions as a plan ; a plan constructed with horizon H is thus p = { a1 , ... , aH } . In a set of plans { p1 , ... , pn } , aij refers to the jth action of the ith plan . We consider models which predict a future state given the current state and one or more actions . We denote R to be the range of such a model , which is the number of steps this model predicts in a single application : fR ( st , at , . . . , at+R−1 ; θ ) = s̃t+R , for parameters θ . We apply the model recursively using the notation FT ( st , at , . . . , aT−1 ; θ ) = fR ( . . . fR ( fR ( st , at , . . . , at+R−1 ; θ ) , at+R , . . . , at+2R−1 ; θ ) . . . , aT−R , . . . , aT−1 ; θ ) = s̃T . That is , FT ( ) applies fR ( ) recursively T/R times . 2.2 PLAN-CONDITIONAL PREDICTORS . To test our conjecture that compounding errors limit the asymptotic performance of existing models on long-horizon RL tasks , we propose plan-conditional predictors ( PCPs ) . A PCP takes the form fR ( st , at , . . . , at+R−1 ) = s̃t+R for some range R > 1 . If R = 1 , then it reduces to the standard approach ( Deisenroth & Rasmussen , 2011 ; Gal et al. , 2016 ; Henaff et al. , 2017 ; Nagabandi et al. , 2017 ; Chua et al. , 2018 ) which predicts only a single time-step at a time , and which we call an action-conditional predictor ( ACP ) . As shown in Figure 1 , a PCP can predict H steps into the future using H/R recursive applications of the model instead of H applications required by an actionconditional predictor . There are many possible parameterizations that could be used in a PCP . In this work we choose deep fully-connected neural networks . There are several reasons for this : ( i ) the space of inputs grows exponentially with the range R , thus models with high capacity are needed to minimize model bias ; ( ii ) since the goal is to understand asymptotic performance , a large data regime is assumed and sample complexity is a secondary issue ; ( iii ) by predicting R steps with a single network application , they are extremely fast at planning time . An obvious alternate parameterization is recurrent neural networks ( RNNs ) , and we provide experimental comparisons between the two approaches in Section 3 . For both Swimmer and HalfCheetah , the ACP and PCP networks consist of fully-connected networks with 9 hidden layers with 1000 units each using the SELU activation function ( Klambauer et al. , 2017 ) . The input state and action ( s ) are concatenated before being used as input to the network . The same loss function is used for training PCPs and ACPs , namely ‖s̃t+R − st+R‖22 , where s is the raw MuJoCo state . We assume that the environment forms a Markov decision process ( MDP ) ( Bellman , 1957 ) with deterministic dynamics , properties shared by the tasks considered in this work . These assumptions allow us to focus exclusively on the significance of model fidelity and planning horizon to MBRL , but removing them is an interesting direction for future work . 2.2.1 INTERMEDIATE PREDICTIONS . For visualization purposes , in some experiments we use a variant of a plan-conditional predictor which takes as input a state and a variable number of actions R′ where 1 ≤ R′ ≤ R , the remainder of action inputs being set to zero . This allows us to plot error or render video of the PCP ’ s predictions at each intermediate timestep instead of only at multiples of R. Furthermore , this variant of the model allows for planning based on reward functions which operate at each timestep rather than just at the end of the episode ( see Section 2.3.1 ) . As such , this variant of the model is applicable to any environment .

Summary: The paper proposes to use a multi-step prediction model in model-based RL. The proposed model maps from current state and a sequence of actions to the state after taking those actions. The paper demonstrates on 2 tasks that in a model-predictive control loop combined with planning by cross-entropy method, this can yield better asymptotic performance than using single-step models.