Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning

1 INTRODUCTION . Both model-based and model-free reinforcement learning ( RL ) methods generally operate in one of two regimes : all training is performed in advance , producing a model or policy that can be used at test-time to make decisions in settings that approximately match those seen during training ; or , training is performed online ( e.g. , as in the case of online temporal-difference learning ) , in which case the agent can slowly modify its behavior as it interacts with the environment . However , in both of these cases , dynamic changes such as failure of a robot ’ s components , encountering a new terrain , environmental factors such as lighting and wind , or other unexpected perturbations , can cause the agent to fail . In contrast , humans can rapidly adapt their behavior to unseen physical perturbations and changes in their dynamics ( Braun et al. , 2009 ) : adults can learn to walk on crutches in just a few seconds , people can adapt almost instantaneously to picking up an object that is unexpectedly heavy , and children that can walk on carpet and grass can quickly figure out how to walk on ice without having to relearn how to walk . How is this possible ? If an agent has encountered a large number of perturbations in the past , it can in principle use that experience to learn how to adapt . In this work , we propose a meta-learning approach for learning online adaptation . Motivated by the ability to tackle real-world applications , we specifically develop a model-based meta-reinforcement learning algorithm . In this setting , data for updating the model is readily available at every timestep in the form of recent experiences . But more crucially , the meta-training process for training such an adaptive model can be much more sample efficient than model-free meta-RL approaches ( Duan et al. , 2016 ; Wang et al. , 2016 ; Finn et al. , 2017 ) . Further , our approach foregoes 1Videos available at : https : //sites.google.com/berkeley.edu/metaadaptivecontrol the episodic framework on which model-free meta-RL approaches rely on , where tasks are pre-defined to be different rewards or environments , and tasks exist at the trajectory level only . Instead , our method considers each timestep to potentially be a new “ task , `` where any detail or setting could have changed at any timestep . This view induces a more general meta-RL problem setting by allowing the notion of a task to represent anything from existing in a different part of the state space , to experiencing disturbances , or attempting to achieve a new goal . Learning to adapt a model alleviates a central challenge of model-based reinforcement learning : the problem of acquiring a global model that is accurate throughout the entire state space . Furthermore , even if it were practical to train a globally accurate dynamics model , the dynamics inherently change as a function of uncontrollable and often unobservable environmental factors , such as those mentioned above . If we have a model that can adapt online , it need not be perfect everywhere a priori . This property has previously been exploited by adaptive control methods ( Åström and Wittenmark , 2013 ; Sastry and Isidori , 1989 ; Pastor et al. , 2011 ; Meier et al. , 2016 ) ; but , scaling such methods to complex tasks and nonlinear systems is exceptionally difficult . Even when working with deep neural networks , which have been used to model complex nonlinear systems ( Kurutach et al. , 2018 ) , it is exceptionally difficult to enable adaptation , since such models typically require large amounts of data and many gradient steps to learn effectively . By specifically training a neural network model to require only a small amount of experience to adapt , we can enable effective online adaptation in complex environments while putting less pressure on needing a perfect global model . The primary contribution of our work is an efficient meta reinforcement learning approach that achieves online adaptation in dynamic environments . To the best knowledge of the authors , this is the first meta-reinforcement learning algorithm to be applied in a real robotic system . Our algorithm efficiently trains a global model that is capable to use its recent experiences to quickly adapt , achieving fast online adaptation in dynamic environments . We evaluate two versions of our approach , recurrence-based adaptive learner ( ReBAL ) and gradient-based adaptive learner ( GrBAL ) on stochastic and simulated continuous control tasks with complex contact dynamics ( Fig . 2 ) . In our experiments , we show a quadrupedal “ ant ” adapting to the failure of different legs , as well as a “ half-cheetah '' robot adapting to the failure off different joints , navigating terrains with different slopes , and walking on floating platforms of varying buoyancy . Our model-based meta RL method attains substantial improvement over prior approaches , including standard model-based methods , online model-adaptive methods , model-free methods , and prior meta-reinforcement learning methods , when trained with similar amounts of data . In all experiments , meta-training across multiple tasks is sample efficient , using only the equivalent of 1.5− 3 hours of real-world experience , roughly 10× less than what model-free methods require to learn a single task . Finally , we demonstrate GrBAL on a real dynamic legged millirobot ( see Fig 2 ) . To highlight not only the sample efficiency of our meta model-based reinforcement learning approach , but also the importance of fast online adaptation in the real world , we show the agent ’ s learned ability to adapt online to tasks such as a missing leg , novel terrains and slopes , miscalibration or errors in pose estimation , and new payloads to be pulled . 2 RELATED WORK . Advances in learning control policies have shown success on numerous complex and high dimensional tasks ( Schulman et al. , 2015 ; Lillicrap et al. , 2015 ; Mnih et al. , 2015 ; Levine et al. , 2016 ; Silver et al. , 2017 ) . While reinforcement learning algorithms provide a framework for learning new tasks , they primarily focus on mastery of individual skills , rather than generalizing and quickly adapting to new scenarios . Furthermore , model-free approaches ( Peters and Schaal , 2008 ) require large amounts of system interaction to learn successful control policies , which often makes them impractical for realworld systems . In contrast , model-based methods attain superior sample efficiency by first learning a model of system dynamics , and then using that model to optimize a policy ( Deisenroth et al. , 2013 ; Lenz et al. , 2015 ; Levine et al. , 2016 ; Nagabandi et al. , 2017b ; Williams et al. , 2017 ) . Our approach alleviates the need to learn a single global model by allowing the model to be adapted automatically to different scenarios online based on recent observations . A key challenge with model-based RL approaches is the difficulty of learning a global model that is accurate for the entire state space . Prior model-based approaches tackled this problem by incorporating model uncertainty using Gaussian Processes ( GPs ) ( Ko and Fox , 2009 ; Deisenroth and Rasmussen , 2011 ; Doerr et al. , 2017 ) . However , these methods make additional assumptions on the system ( such as smoothness ) , and does not scale to high dimensional environments . Chua et al . ( 2018 ) has recently showed that neural networks models can also benefit from incorporating uncertainty , and it can lead to model-based methods that attain model-free performance with a significant reduction on sample complexity . Our approach is orthogonal to theirs , and can benefit from incorporating such uncertainty . Prior online adaptation approaches ( Tanaskovic et al. , 2013 ; Aswani et al. , 2012 ) have aimed to learn an approximate global model and then adapt it at test time . Dynamic evaluation algorithms ( Rei , 2015 ; Krause et al. , 2017 ; 2016 ; Fortunato et al. , 2017 ) , for example , learn an approximate global distribution at training time and adapt those model parameters at test time to fit the current local distribution via gradient descent . There exists extensive prior work on online adaptation in modelbased reinforcement learning and adaptive control ( Sastry and Isidori , 1989 ) . In contrast from inverse model adaptation ( Kelouwani et al. , 2012 ; Underwood and Husain , 2010 ; Pastor et al. , 2011 ; Meier et al. , 2016 ; Meier and Schaal , 2016 ; Rai et al. , 2017 ) , we are concerned in the problem of adapting the forward model , closely related to online system identification ( Manganiello et al. , 2014 ) . Work in model adaptation ( Levine and Koltun , 2013 ; Gu et al. , 2016 ; Fu et al. , 2015 ; Weinstein and Botvinick , 2017 ) has shown that a perfect global model is not necessary , and prior knowledge can be fine-tuned to handle small changes . These methods , however , face a mismatch between what the model is trained for and how it is used at test time . In this paper , we bridge this gap by explicitly training a model for fast and effective adaptation . As a result , our model achieves more effective adaptation compared to these prior works , as validated in our experiments . Our problem setting relates to meta-learning , a long-standing problem of interest in machine learning that is concerned with enabling artificial agents to efficiently learn new tasks by learning to learn ( Thrun and Pratt , 1998 ; Schmidhuber and Huber , 1991 ; Naik and Mammone , 1992 ; Lake et al. , 2015 ) . A meta-learner can control learning through approaches such as deciding the learner ’ s architecture ( Baker et al. , 2016 ) , or by prescribing an optimization algorithm or update rule for the learner ( Bengio et al. , 1990 ; Schmidhuber , 1992 ; Younger et al. , 2001 ; Andrychowicz et al. , 2016 ; Li and Malik , 2016 ; Ravi and Larochelle , 2018 ) . Another popular meta-learning approach involves simply unrolling a recurrent neural network ( RNN ) that ingests the data ( Santoro et al. , 2016 ; Munkhdalai and Yu , 2017 ; Munkhdalai et al. , 2017 ; Mishra et al. , 2017 ) and learns internal representations of the algorithms themselves , one instantiation of our approach ( ReBAL ) builds on top of these methods . On the other hand , the other instantiation of our method ( GrBAL ) builds on top of MAML ( Finn et al. , 2017 ) . GrBAL differs from the supervised version of MAML in that MAML assumes access to a hand-designed distribution of tasks . Instead , one of our primary contributions is the online formulation of meta-learning , where tasks correspond to temporal segments , enabling “ tasks ” to be constructed automatically from the experience in the environment . Meta-learning in the context of reinforcement learning has largely focused on model-free approaches ( Duan et al. , 2016 ; Wang et al. , 2016 ; Sung et al. , 2017 ; Al-Shedivat et al. , 2017 ) . However , these algorithms present even more ( meta- ) training sample complexity than non-meta model-free RL methods , which precludes them from real-world applications . Recent work ( Sæmundsson et al. , 2018 ) has developed a model-based meta RL algorithm , framing meta-learning as a hierarchical latent variable model , training for episodic adaptation to dynamics changes ; the modeling is done with GPs , and results are shown on the cart-pole and double-pendulum agents . In contrast , we propose an approach for learning online adaptation of high-capacity neural network dynamics models ; we present two instantiations of this general approach and show results on both simulated agents and a real legged robot .

Summary: This work addresses the problem of online adapting dynamics models in the context of model-based RL. Learning globally accurate dynamics model is impossible if we consider that environments are dynamic and we can't observe every possible environment state at initial training time. Thus learning dynamics models that can be adapted online fast, to deal with unexpected und never seen before events is an important research problem.