Model-Based Offline Planning

1 INTRODUCTION . Learnt policies for robotic and industrial systems have the potential to both increase existing systems ’ efficiency & robustness , as well as open possibilities for systems previously considered too complex to control . Learnt policies also afford the possibility for non-experts to program controllers for systems that would currently require weeks of specialized work . Currently , however , most approaches for learning controllers require significant interactive time with a system to be able to converge to a performant policy . This is often either undesirable or impossible due to operating cost , safety issues , or system availability . Fortunately , many systems are designed to log sufficient data about their state and control choices to create a dataset of operator commands and resulting system states . In these cases , controllers could be learned offline , using algorithms that produce a good controller using only these logs , without ever interacting with the system . In this paper we propose such an algorithm , which we call Model-Based Offline Planning ( MBOP ) , which is able to learn policies directly from logs of a semi-performant controller without interacting with the corresponding environment . It is able to leverage these logs to generate a more performant policy than the one used to generate the logs , which can subsequently be goal-conditioned or constrained dynamically during system operation . Learning from logs of a system is often called ‘ Offline Reinforcement Learning ’ ( Wu et al. , 2019 ; Peng et al. , 2019 ; Fujimoto et al. , 2019 ; Wang et al. , 2020 ) and both model-free ( Wu et al. , 2019 ; Wang et al. , 2020 ; Fujimoto et al. , 2019 ; Peng et al. , 2019 ) and model-based ( Yu et al. , 2020 ; Kidambi et al. , 2020 ) approaches have been proposed to learn policies in this setting . Current modelbased approaches , MOPO ( Yu et al. , 2020 ) and MoREL ( Kidambi et al. , 2020 ) , learn a model to train a model-free policy in a Dyna-like ( Sutton & Barto , 2018 ) manner . Our proposed approach , MBOP , is a model-based approach that leverages Model-Predictive Control ( MPC ) ( Rault et al. , 1978 ) and extends the MPPI ( Williams et al. , 2017b ) trajectory optimizer to provide a goal or reward-conditioned policy using real-time planning . It combines three main elements : a learnt world model , a learnt behavior-cloning policy , and a learnt fixed-horizon value-function . MBOP ’ s key advantages are its data-efficiency and adaptability . MBOP is able to learn policies that perform better than the demonstration data from as little as 100 seconds of simulated system time ( equivalent to 5000 steps ) . A single trained MBOP policy can be conditioned with a reward function , a goal state , as well as state-based constraints , all of which can be non-stationary , allowing for easy control by a human operator or a hierarchical system . Given these two key advantages , we believe it to be a good candidate for real-world use in control systems with offline data . We contextualize MBOP relative to existing work in Section 2 , and describe MBOP in Section 3 . In Section 4.2 , we demonstrate MBOP ’ s performance on standard benchmark performance tasks for offline RL , and in Section 4.3 we demonstrate MBOP ’ s performance in zero-shot adaptation to varying task goals and constraints . In Section 4.4 we perform an ablation analysis and consider combined contributions of MBOP ’ s various elements . 2 RELATED WORKS . Model-Based approaches with neural networks have shown promising results in recent years . Guided Policy Search ( Levine & Koltun , 2013 ) leverages differential dynamic programming as a trajectory optimizer on locally linear models , and caches the resulting piece-wise policy in a neural network . Williams et al . ( 2017b ) show that a simple model-based controller can quickly learn to drive a vehicle on a dirt track , the BADGR robot ( Kahn et al. , 2020 ) also uses Model-Predictive Path Integral ( MPPI ) ( Williams et al. , 2017a ) with a learned model to learn to navigate to novel locations , Yang et al . ( 2020 ) show good results learning legged locomotion policies using MPC with learned models , and ( Ebert et al. , 2018 ) demonstrate flexible robot arm controllers leveraging learned models with image-based goals . Silver et al . ( 2016 ) have shown the power of additional explicit planning in various board games including Go . More recently planning-based algorithms such as PlaNet ( Hafner et al. , 2019b ) have shown strong results in pixel-based continuous control tasks by leveraging latent variational RNNs . Simpler approaches such as PDDM ( Nagabandi et al. , 2020 ) or PETS ( Chua et al. , 2018 ) have shown good results using full state information both in simulation and on real robots . MBOP is strongly influenced by PDDM ( Nagabandi et al. , 2020 ) ( itself an extension on PETS ( Chua et al. , 2018 ) ) , in particular with the use of ensembles and how they are leveraged during planning . PDDM was not designed for offline use , and MBOP adds a value function composition as well as a policy prior during planning to increase data efficiency and strengthen the set of priors for offline learning . It leverages the same trajectory re-weighting approach used in PDDM and takes advantage of its beta-mixture of the T trajectory buffer . Both MoREL ( Kidambi et al. , 2020 ) and MOPO ( Yu et al. , 2020 ) leverage model-based approaches for offline learning . This is similar to approaches used in MBPO ( Janner et al. , 2019 ) and DREAMER ( Hafner et al. , 2019a ) , both of which leverage a learnt model to learn a model-free controller . MoREL and MOPO , however , due to their offline nature , train their model-free learner by using a surrogate MDP which penalizes for underlying model uncertainty . They do not use the models for direct planning on the problem , thus making the final policy task-specific . MOPO demonstrate the ability of their algorithm to alter the reward function and re-train a new policy according to this reward , but can not leverage the final policy to dynamically adapt to an arbitrary goal or constrained objective . Matsushima et al . ( 2020 ) use a model-based policy for deployment efficient RL . Their use case is a mix between offline and online RL , where they consider that there is a limited number of deployments . They share a similarity in the sense that they also use a behaviorcloning policy πβ to guide trajectories in a learned ensemble model , but perform policy improvement steps on a parametrized policy initialized from πβ using a behavior-regularized objective function . Similarly to MoREL and MOPO their approach learns a parameterized policy for acting in the real system . The use of a value function to extend the planning horizon of a planning-based policy has been previously proposed by Lowrey et al . ( 2018 ) with the POLO algorithm . POLO uses a ground-truth model ( e.g . physics simulator ) with MPPI/MPC for trajectory optimization . POLO additionally learns an approximate value-function through interaction with the environment which is then appended to optimized trajectories to improve return estimation . Aside from the fact that MBOP uses an entirely approximate & learned model , it uses a similar idea but with a fixed-horizon value function to avoid bootstrapping , and separate heads of the ensemble during trajectory optimization . BC-trained policies as sampling priors have been looked at by POPLIN ( Wang & Ba , 2019 ) . POPLIN does not use value bootstrapping , and re-samples an ensemble head at each timestep during rollouts , which likely provides less consistent variations in simulated plans . They show strong results relative to a series of model-based and model-free approaches , but do not manage to perform on the Gym Walker envi- ronment . Additionally , they are overall much less data efficient than MBOP and do not demonstrate performance in the offline setting . Task-time adaptation using model-based approaches has been considered previously in the modelbased literature . Lu et al . ( 2019 ) look at mixing model-free and model-based approaches using notions of uncertainty to allow for adaptive controllers for non-stationary problems . Rajeswaran et al . ( 2020 ) use a game-theoretic framework to describe two adaptive learners that are both more sample efficient than common MBRL algorithms , as well as being more robust to non-stationary goals and system dynamics . MBOP is able to perform zero-shot adaptation to non-stationary goals and constraints , but does not provide a mechanism for dealing with non-stationary dynamics . If brought into the on-line settings , approaches from these algorithms such as concentrating on recent data , could however be leveraged to allow for this . Previous approaches all look at various elements present in MBOP but none consider the full combination of a BC prior on the trajectory optimizer with a value-function initialization , especially in the case of full offline learning . Along with this high-level design , many implementation details such as consistent ensemble sampling during rollouts , or averaging returns over ensemble heads , appear to be important for a stable controller from our experience . 3 MODEL-BASED OFFLINE PLANNING . Our proposed algorithm , MBOP ( Model-Based Offline Planning ) , is a model-based RL algorithm able to produce performant policies entirely from logs of a less-performant policy , without ever interacting with the actual environment . MBOP learns a world model and leverages a particle-based trajectory optimizer and model-predictive control ( MPC ) to produce a control action conditioned on the current state . It can be seen as an extension of PDDM ( Nagabandi et al. , 2020 ) , with a behaviorcloned policy used as a prior on action sampling , and a fixed-horizon value function used to extend the planning horizon . In this following sections , we introduce the Markov Decision Process ( MDP ) formalism , briefly explain planning-based approaches , discuss offline learning , and then introduce the elements of MBOP before describing the algorithm in full . 3.1 MARKOV DECISION PROCESS . Let us model our tasks as a Markov Decision Process ( MDP ) , which can be defined as a tuple ( S , A , p , r , γ ) , where an agent is in a state st ∈ S and takes an action at ∈ A at timestep t. When in state st and taking an action at , an agent will arrive in a new state st+1 with probability p ( st+1|st , at ) , and receive a reward r ( st , at , st+1 ) . The cumulative reward over a full episode is called the return R and can be truncated to a specific horizon as RH . Generally reinforcement learning and control aim to provide an optimal policy function πs : S → A which will provide an action at in state st which will lead to the highest long-term return : π∗ ( st ) = argmaxa∈A ∑∞ t=1 γ tr ( st , π ∗ ( st ) ) , where γ is a time-wise discounting factor that we fix to γ = 1 , and therefore only consider finite-horizon returns . 3.2 PLANNING WITH LEARNED MODELS . A large body of the contemporary work with MDPs involves Reinforcement Learning ( RL ) Sutton & Barto ( 2018 ) with model-free policies Mnih et al . ( 2015 ) ; Lillicrap et al . ( 2015 ) ; Schulman et al . ( 2017 ) ; Abdolmaleki et al . ( 2018 ) . These approaches learn some form of policy network which provides its approximation of the best action at for a given state st often as a single forward-pass of the network . MBOP and other model-based approaches Deisenroth & Rasmussen ( 2011 ) ; Chua et al . ( 2018 ) ; Williams et al . ( 2017b ) ; Hafner et al . ( 2019b ) ; Lowrey et al . ( 2018 ) ; Nagabandi et al . ( 2020 ) are very different . They learn an approximate model of their environment and then use a planning algorithm to find a high-return trajectory through this model , which is then applied to the environment 1 . This is interesting because the final policy can be more easily adapted to new 1This approach is often called Model-Based Reinforcement Learning ( MBRL ) in the literature , but we chose to talk more generally about planning with learned models as the presence of a reward is not fundamentally necessary and the notion of reinforcement is much less present . tasks , be made to respect constraints , or offer some level of explainability . When bringing learned controllers to industrial systems , many of these aspects are highly desireable , even to the expense of raw performance .

Summary: This work studies the offline RL problem and proposes MBOP for the same. The proposed method learns ensembles of dynamics models, behavioral policies, and value functions using the offline dataset. Subsequently, the approach uses online MPC with a learned terminal value function. The paper demonstrates experimental results on standard benchmark tasks (RLU and D4RL) as well as zero-shot adaptation results.