Hope For The Best But Prepare For The Worst: Cautious Adaptation In RL Agents

We study the problem of safe adaptation : given a model trained on a variety of past experiences for some task , can this model learn to perform that task in a new situation while avoiding catastrophic failure ? This problem setting occurs frequently in real-world reinforcement learning scenarios such as a vehicle adapting to drive in a new city , or a robotic drone adapting a policy trained only in simulation . While learning without catastrophic failures is exceptionally difficult , prior experience can allow us to learn models that make this much easier . These models might not directly transfer to new settings , but can enable cautious adaptation that is substantially safer than naı̈ve adaptation as well as learning from scratch . Building on this intuition , we propose risk-averse domain adaptation ( RADA ) . RADA works in two steps : it first trains probabilistic model-based RL agents in a population of source domains to gain experience and capture epistemic uncertainty about the environment dynamics . Then , when dropped into a new environment , it employs a pessimistic exploration policy , selecting actions that have the best worst-case performance as forecasted by the probabilistic model . We show that this simple maximin policy accelerates domain adaptation in a safety-critical driving environment with varying vehicle sizes . We compare our approach against other approaches for adapting to new environments , including meta-reinforcement learning . 1 INTRODUCTION . An experienced human driving a rental car for the first time is initially very aware of her lack of familiarity with the car . How sensitive is it to acceleration and braking ? How does it respond to steering ? How wide is the vehicle and what is its turning radius ? She drives mindfully , at low speeds , braking far ahead of desired stops , and making wide turns , all the while observing the car ’ s responses and adapting to it . Within minutes , once she is familiar with the car , she begins to drive more fluently and efficiently . Humans draw upon their prior experiences to perform this kind of safe , quick adaptation to unfamiliar situations all the time , such as when playing with a new tennis racquet , or walking on a new slippery surface . Such problems are critical to address in autonomous systems : such as when a self-driving car must learn to drive in a new country , or when a planetary rover might have to learn to explore a harsh new environment . Missteps in real-world situations can cause real damage to robots and their environments . An important bottleneck in applying today ’ s standard machine learning approaches to control in these real-world situations is that they are trained without any notion of safe behavior under uncertainty . Recent works have attempted to address this by proposing methods for safe exploration during reinforcement learning — in other words , how might an agent avoid risky actions during training time ? This still requires that the robot acquire its notions of uncertainty and risks at the same time as it is learning to perform tasks in the new environment , which is difficult and precarious . Could we instead rely on transferring notions of uncertainty and risk acquired from prior experience in other related domains , such as in simulated environments , where safety may not be as much of a concern ? In other words , could we make the safe learning problem easier through knowledge transfer , relaxing the problem to safe adaptation , like the human driver ? How might the planetary rover draw on its experience in many varied terrains on Earth to perform meaningfully cautious actions during learning on the unknown terrain of a new planet ? Motivated by these questions , we propose a model-based reinforcement learning approach called risk averse domain adaptation ( RADA ) . RADA works by first pretraining a probabilistic dynamics model on a population of training domains with varied , unknown dynamics . Through this experience over many environments , the model learns to estimate the epistemic uncertainty ( model uncertainty ) of unknown environment dynamics , thus permitting estimation of a distribution of outcomes for any action executed by the agent . When introduced into a new target environment , RADA uses this estimated distribution of outcomes to select cautious actions that obey the following maximin notion of risk-aversion : among various candidate action sequences , it executes those that lead to the best worst-case performance , as predicted by the model . Much like the human driver in the example above , all the information collected during this cautious phase of exploration is fed back into the model to finetune it to the new domain , leading to increasingly confident predictions . Over time , RADA steadily estimates lower risks and approaches optimality in the target environment . As we demonstrate in experiments in a driving domain , the experience acquired during RADA ’ s pretraining phase enables fast yet safe adaptation within only a handful of episodes . 2 RELATED WORK . Cautious or risk-averse learning has close connections to learning robust control policies , as well as the uncertainty estimation derived from Bayesian reinforcement learning ( Ghavamzadeh et al. , 2015 ; Strens , 2000 ) . Rather than conventionally maximizing a reward function , accounting for risk usually involves allocating more attention to ‘ worst-case ’ outcomes in an environment . Such outcomes become particularly important in out-of-domain settings , where purely optimizing in the training domain does not guarantee good performance in the test domain , the problem setting that we consider in this work . Safety in reinforcement learning . Incorporating safety requires properly managing risks and reducing the impact of unforeseen negative outcomes . Risk management is extensively studied in quantitative finance . In portfolio optimization , a commonly used quantity that measures the expected return considering the worse α- % of cases is Conditional Value at Risk ( CVaR ) ( Rockafellar et al. , 2000 ) . With probability α , the reward is greater than the CVaR measure . CVaR is formulated as E [ R|R ≤ υα ] . Rather than optimizing the expected reward , risk averse policies optimize the lower α-quartile of the distribution of rewards . While meta-learning approaches like RL2 ( Duan et al. , 2016 ) can potentially learn safety by adapting across learning episodes , we found this was not possible in the environments we tested . To address safety more expicitly , the reinforcement learning community is adopting measures like CVaR as quantities that can be optimized ( Morimura et al. , 2010 ; Borkar & Jain , 2010 ; Chow & Ghavamzadeh , 2014 ; Tamar et al. , 2015 ; Chow et al. , 2015 ) to create policies which are robust to shifts from source to target domains . Rajeswaran et al . ( 2016 ) propose learning robust policies by sampling from the α-quartile of an ensemble of models . While the model ensemble is trained on a given source distribution , the policy is only trained on the lower α-quartile rewards from trajectories sampled from this ensemble . This leads to policies which are more conservative and therefore more robust to shift when deployed from source to target domains . Epistemic uncertainty in reinforcement learning . While learning a robust model is beneficial for transferring to different domains , model-based reinforcement learning offers an additional unsupervised learning signal that can be exploited at test time . In particular , prior work has shown that a model can be quickly adapted during test time by meta-learning for fast adapting parameters during training ( Nagabandi et al. , 2018a ; Sæmundsson et al. , 2018 ) . These fast adapting parameters offers greater flexibility in adapting to unforeseen circumstances which an agent may encounter at test time . Nagabandi et al . ( 2018a ) show that real robots can quickly adapt to broken or miscalibrations during evaluation through this fast adaptation acquired through meta-learning . Such approaches are complementary to our approach , as they provide a means to explicitly train for fast adaptation to disturbances in the environment , while they do not account for any notion of safety . Henaff et al . ( 2019 ) propose using the uncertainty of a model to regularize policy learning . The policy is encouraged to create trajectories which are distributionally close to trajectories observed in training data . After training , the observations of the model and actions of the policy generate state trajectories which near the domain it has been trained on . In other words , the policy has a preference to keep the trajectories within its training domain . In our work , our policy is encouraged to behave cautiously in unfamiliar environments rather than remain in familiar ones . Kahn et al . ( 2017 ) train a collision prediction model to favor ’ safe ’ ( low velocity ) collisions . Using uncertainty estimates , this collision model will initially behave cautiously in a new environment . Similar to our method , the model becomes less conservative as it adapts to the new environment and lowers its uncertainty . Domain randomization . Domain randomization ( Sadeghi & Levine , 2017 ; Peng et al. , 2018 ; Tobin et al. , 2017 ) attempts to train policies that are able to transfer to some target environment by training an agent in deliberately randomized simulated environments to allow learning a policy that is invariant to the randomized parameters , and thus performs robustly in the target environment . RADA also pretrains on a set of environments with varied dynamics , but different from these prior works , we operate in a safety-critical setting , focusing on safe adaptation to the target environment — to accomplish this , we follow an explicitly cautious action policy at adaptation time , different from the policy used in the pretraining environments . Before discussing RADA , we first lay out some preliminaries . 3 BACKGROUND : PETS . We build upon PETS ( Chua et al. , 2018 ) , a recently proposed approach for model-based reinforcement learning . We describe the main features of the PETS framework below : Probabilistic dynamics model . PETS trains an ensemble of probabilistic dynamics models within its environment . Each model in the ensemble is a probabilistic neural network that outputs a distribution over the next state s′ conditioned on the current state s and action a . The data for training these models comes from trajectories executed by following the same scheme for action selection that will be eventually used at test time . Action selection . This action selection scheme is sampling-based model-predictive control ( MPC ) : an evolutionary search method finds action sequences with the highest predicted reward . The reward of an action sequence in turn is computed by propagating action outcomes autoregressively through the learned probabilistic models . Reward computation . Specifically , starting from a state s0 , for each sampled action sequence A = [ a1 , ... , aH ] , where H is the planning horizon , the dynamics model first predicts a distribution over s1 after executing a0 . A particle propagation method samples Monte Carlo samples from this distribution . For each sample , the dynamics model then predicts the state distribution for s2 , conditioned on executing a1 , and the process repeats . This recursive particle propagation results in a large number N of particles { ŝiH } Ni=1 after H steps . These N particles represent samples from the distribution of possible states after executing A . Each such particle i ∈ [ 1 , N ] is now assigned a predicted reward ri , which is a function of its full state trajectory starting from s0 . Finally , the mean of those predicted rewards is considered the score of the action sequence : R ( A ) = ∑ i ri/N . ( 1 ) We call this the action score . Then , the winning action sequence A∗ = argmaxAR ( A ) with the highest action score is selected , the first action in that sequence is executed , and the whole process repeats starting from the resulting new state s1 .

Summary: This paper studies the problem of safe adaptation to avoid catastrophic failure in a new environment. It draws intuition from human behavior. The proposed method (risk-averse domain adaptation (RADA)) learns probabilistic model-based RL agents from source domains, and uses them to select actions that has the best worst-case performance in the target domain.