Hope For The Best But Prepare For The Worst: Cautious Adaptation In RL Agents
Authors: Jesse Zhang
Brian Cheung
Chelsea Finn
Dinesh Jayaraman
Sergey Levine
Venue:  (2020)
Keywords: safety
risk
uncertainty
adaptation

Abstract: We study the problem of safe adaptation: given a model trained on a variety of past experiences for some task, can this model learn to perform that task in a new situation while avoiding catastrophic failure? This problem setting occurs frequently in real-world reinforcement learning scenarios such as a vehicle adapting to drive in a new city, or a robotic drone adapting a policy trained only in simulation. While learning without catastrophic failures is exceptionally difficult, prior experience can allow us to learn models that make this much easier. These models might not directly transfer to new settings, but can enable cautious adaptation that is substantially safer than na\"{i}ve adaptation as well as learning from scratch. Building on this intuition, we propose risk-averse domain adaptation (RADA). RADA works in two steps: it first trains probabilistic model-based RL agents in a population of source domains to gain experience and capture epistemic uncertainty about the environment dynamics. Then, when dropped into a new environment, it employs a pessimistic exploration policy, selecting actions that have the best worst-case performance as forecasted by the probabilistic model. We show that this simple maximin policy accelerates domain adaptation in a safety-critical driving environment with varying vehicle sizes. We compare our approach against other approaches for adapting to new environments, including meta-reinforcement learning.

Under review as a conference paper at ICLR 2020
HOPE FOR THE BEST BUT PREPARE FOR THE WORST: CAUTIOUS ADAPTATION IN RL AGENTS
Anonymous authors Paper under double-blind review
ABSTRACT
We study the problem of safe adaptation: given a model trained on a variety of past experiences for some task, this model learn to perform that task in new situa- tion while avoiding catastrophic failure? This problem setting occurs frequently in real-world reinforcement learning scenarios such as vehicle adapting to drive in a new city, or a robotic drone adapting a policy trained only in simulation. While learning without catastrophic failures is exceptionally difficult, prior experience can allow us to learn models that make this much easier. These models might not directly transfer to new settings, but can enable cautious adaptation that is substantially safer than na've adaptation as well as learning from scratch. Building on this intuition, we propose risk-averse domain adaptation (RADA). RADA works in two steps: it first trains probabilistic model-based RL agents in a population of source domains to gain experience and capture epistemic uncertainty about the environment dynamics. Then, when dropped into pessimistic exploration policy, selecting actions that have the best worst-case perfor- mance forecasted by the probabilistic model. We show that this simple maximin policy accelerates domain adaptation in a safety-critical driving environment with varying vehicle sizes. We compare approach against other approaches for adapting to new environments, including meta-reinforcemen learning.
environment, it employs
INTRODUCTION
An experienced human driving a rental car for the first time is initially very aware of her lack of familiarity with the car. How sensitive is it to acceleration and braking? How does it respond to steering? How wide is the vehicle and what is its turning radius? She drives mindfully, at low speeds, braking far ahead of desired stops, and making wide turns, all the while observing the car's s responses and adapting to it. Within minutes, once she is familiar with the car, she begins to drive more fluently and efficiently. Humans draw upon their prior experiences perform this kind of safe, quick adaptation to unfamiliar situations all the time, such as when playing with a new tennis racquet, or walking new slippery surface.
Such problems are critical to address in autonomous systems: such as when a self-driving car must learn drive in new country, or when planetary rover might have to learn to explore harsh new environment. Missteps in real-world situations can cause real damage to robots and their environments. An important bottleneck in applying today's standard machine learning approaches to control in these real-world situations is that they are trained without any notion of safe behavior under uncertainty. Recent works have attempted to address this by proposing methods for safe exploration during reinforcement learning training time? This still requires that the robot acquire its notions of uncertainty and risks at the same time it is learning to perform tasks in the new environment, which is difficult and precarious.
in other words, how might an agent avoid risky actions during
Could we instead rely on transferring notions uncertainty and risk acquired from prior experience in other related domains, such in simulated environments, where safety may not be as much of concern? In other words, could we make the safe learning problem easier through knowledge transfer, relaxing the problem to safe adaptation, like the human driver? How might the planetary rover draw on its experience in many varied terrains on Earth to perform meaningfully cautious actions during learning the unknown terrain of new planet?

---

Under review as a conference paper at ICLR 2020
Motivated by these questions, we propose model-based reinforcement learning approach called risk averse domain adaptation (RADA). RADA works by first pretraining a probabilistic dynamics model on a population of training domains with varied, unknown dynamics. Through this experience over many environments, the model learns to estimate the epistemic uncertainty (model uncertainty) of unknown environment dynamics, thus permitting estimation of distribution of outcomes for any action executed by the agent. When introduced into a new target environment, RADA uses this estimated distribution of outcomes select cautious actions that obey the following maximin notion of aveesinosoon: among various candidate action sequences, it executes those that lead to the best worst-case performance, as predicted by the model. Much like the human driver in the example above, all the information collected during this cautious phase of exploration is fed back into the model to finetune to the new domain, leading to increasingly confident predictions. Over time, RADA steadily estimates lower risks and approaches optimality in the target environment. As we demonstrate in experiments in a driving domain, the experience acquired during RADA's pretraining phase enables fast yet safe adaptation within only handful of episodes.
2 RELATED WORK
Cautious or risk-averse learning has close connections to learning robust control policies, as well as the uncertainty estimation derived from Bayesian reinforcement learning (Ghavamzadeh et al. 2015] Strens 2000). Rather than conventionally maximizing a reward function, accounting for risk usually involves allocating more attention to 'worst-case' outcomes in an environment. Such outcomes become particularly important in out-of-domain settings, where purely optimizing in the training domain does not guarantee good performance in the test domain, the problem setting that we consider in this work.
Safety in reinforcement learning. Incorporating safety requires properly managing risks and reducing the impact of unforeseen negative outcomes. Risk management is extensively studied in quantitative finance. In portfolio optimization, a commonly used quantity that measures the expected return considering the worse a-% of cases Conditional Value at Risk (CVaR) (Rockafellar et al.) 2000). With probability a, the reward is greater than the CVaR measure. CVaR is formulated as E S val. Rather than optimizing the expected reward, risk averse policies optimize the lower a-quartile of the distribution of rewards.
While meta-learnigg approaches like RL3 DDuan et al. 2016 can potentially learn safety by adapting across learning episodes, we found this was not possible in the environments we tested. To address safety more expicitly, the reinforcement learning community is adopting measures like CVaR as quantities that can be optimized (Morimura et al. 2010 Borkar & Jain 2010: Chow & Ghavamzadehi 2014: Tamar et al., 2015 Chow et al. (0155) [ to create policies which are robust to shifts from source to target domains. Rajeswaran et al. (016)) propose learning robust policies by sampling from the a-quartile of an ensemble of models. While the model ensemble is trained on a given source distribution, the policy is only trained on the lower a-qurrtile rewards from trajectories sampled from this ensemble. This leads to policies which are more conservative and therefore more robust to shift when deployed from source target domains.
Epistemic uncertainty in reinforcement learning. While learning a robust model is beneficial for transferring different domains, model-based reinforcement learning offers an additional unsuper- vised learning signal that can be exploited at test time. In particular, prior work has shown that a model can be quickly adapted during test time by meta-learning for fast adapting parameters during training (Nagabandi et al.) 2018a Saemundsson et al.) 2018). These fast adapting parameters offers greater flexibility adapting to unforeseen circumstances which an agent may encounter test time. Nagabandi et al. (20T8a) 8a) show that real robots can quickly adapt to broken or miscalibrations during evaluation through this fast adaptation acquired through meta-learning. Such approaches are complementary to approach, as they provide means to explicitly train for fast adaptation to disturbances in the environment, while they do not account for any notion of safety.
Henaff et al. (0019) propose using the uncertainty of model to regularize policy learning. The policy is encouraged to create trajectories which are distributionally close to trajectories observed in training data. After training, the observations of the model and actions of the policy generate state trajectories which near the domain it has been trained on. In other words, the policy has preference to keep the trajectories within its training domain. In our work, our policy is encouraged to behave
2

---

Under review as a conference paper at ICLR 2020
cautiously in unfamiliar environments rather than remain in familiar ones. Kahn et al. 2017) train collision prediction model to favor 'safe' (low velocity) collisions. Using uncertaityy estimates, this collision model will initially behave cautiously in environment. Similar to our method, the model becomes less conservative as it adapts to the new environment and lowers its uncertainty.
Domain randomization. Domain randomization (Sadeghi & Levine, 2017 Peng et al. 2018
Tobin et al., (017) attempts to train policies that are able to transfer to some target environment by training agent deliberately randomized simulated environments to allow learning policy that is invariant to the randomized parameters, and thus performs robustly in the target environment. RADA also pretrains on set of environments with varied dynamics, but different from these prior works, we operate in safety-critical setting, focusing safe adaptation to the target environment - to accomplish this, we follow an explicitly cautious action policy at adaptation time, different from the policy used in the pretraining environments.
Before discussing RADA, we first lay out some preliminaries.
BACKGROUND: PETS
We build upon PETS (Chua et al. 2018), a recently proposed approach for model-based reinforcement learning. We describe the main features of the PETS framework below:
Probabilistic dynamics model. PETS trains an ensemble of probabilistic dynamics models within its environment. Each model in the ensemble is probabilistic neural network that outputs distribution over the next state conditioned on the current state and action a. The data for training these models comes from trajectories executed by following the same scheme for action selection that will be eventually used at test time.
Action selection. This action selection scheme is sampling-based model-predictive control (MPC): an evolutionary search method finds action sequences with the highest predicted reward. The reward of action sequence in turn is computed by propagating action outcomes autoregressively through the learned probabilistic models.
Specifically, starting from state 8o. for each sampled action sequence A = =[a....,, aH), where H is the planning horizon, the dynamics model first predicts a distribution $1 after executing ag. A particle propagation method samples Monte Carlo samples from this distribution. For each sample, the dynamics model then predicts the state distribution for 82, conditioned on executing d, and the process repeats. This recursive particle propagation results in a large number N of particles {8), 1 after H steps. These N particles represent samples from the distribution of possible states after executing A. Each such particle E [1, .N) is now assigned predicted reward r', which is a function of its full state trajectory starting from Sp- Finally, the mean of those predicted rewards is considered the score of the action sequence:
Reward computation.
R(A) = Σ r ///
We call this the action score. Then, the winning action sequence A = arg max, R(A) with the highest action score is selected, the first action in that sequence executed, and the whole process repeats starting from the resulting new state
4 RADA: RISK-AVERSE DOMAIN ADAPTATION
Now we present our approach, Rier-everse Domain Adaptation (RADA). As motivated in Sec [ RADA approaches safe learning as an adaptation problem, where an agent may draw upon its experience in variety of environments to guide cautious behavior in a new safety-critical target environment while minimizing the risk of catastrophic failure.
Consider a set of environments, each defined by the value of an unknown domain ID variable which controls the dynamics in each domain. RADA assumes a setting where we train on some of these domains, and must then transfer new domains with unknown, potentially unseen, and even out-of- disttiibbttiitoiinnn values of z. As a rumning example, consider learning to drive cars, where each is represented by its own value of the domain IID variable This domain ID might include
3

---

Under review as a conference paper at ICLR 2020
potentially large set of unknown and hard-to-menn no-mmasure properties, which together determine the way that the car drives.
We propose solution, RADA, that builds upon the PETS framework (Sec 3/. PETS has been demonstrated to work well across a variety of environments. Further, compared to alternative approaches, this framework has two natural advantages for our risk-averse domain adaptation setting, both which are critical to our approach: (i) the probabilistic models of PETS can be adapted to capture the "epistemic uncertainty" about the dynamics of new domain, and (ii) model-based RL agents contain dynamics models that can be trained in the absence of any rewards or supervision, providing a route for adaptation new environment. We now discuss how RADA builds upon PETS.
RADA first builds a probabilistic ensemble dynamics models on the training domains that capture the epistemic uncertainty in dynamics due to unknown We call this the pretraining phase. When an agent with this pretrained model encounters a new domain, we use pessimistic predictions from the model to drive cautious exploration finetune the model to the new domain, leading to safe and fast adaptation. We call this the adaptation/finttning phase. Algorithm provides pseudocode for RADA, and the rest of this section explains RADA in detail.
4.1 PRETRAINING PETS IN MULTIPLE DOMAINS
While the PETS probabilistic ensemble trained to represent uncertainty within a single environment in Chua et al. 2018), would like model to capture the uncertainty associated with being dropped into a new environment, with unknown domain ID 2.
To do this, propose "pretraining" phase, where single PETS ensemble is trained across all the training environments, with varying, unknown values of z. Specifically, at the beginning of each training episode, we randomly sample one of the training z's from uniform distribution. Since determines the environment dynamics and is unavailable to the learned dynamics model, the ensemble has incentive learn model this epistemic uncertainty during the pretraining phase. See the pretraining procedure in Algorithm
4.2 CAUTIOUS ACTION SELECTION DURING ADAPTATION
After this pretraining, how might the uncertainty captured the ensemble inform cautious exploration during adaptation in the target environment? To do this, we adapt the PETS action selection and reward computation scheme using a maximin notion of cautious behavior, in line with notions risk
used in prior work across disciplines (Rockafellar et al. 2000 Tamar et al. 2015 Rajeswaran et al.) 2016).
Specifically, replace the action of equation I with newly defined "generalized action score" R., (A), which the "caution parameter" 7 E [0, I00] controls the degree of caution exercised in evaluating action sequences in the environment. R.y (A) is defined as:
(A) =
Σ Scr (1n0--y)))
*/N,
(2)
where (r)r) denotes the value of the 1 percentile of predicted rewards ) among the N particles after particle propagation.
Unpacking this definition, R., measures the mean score of the bottom 100 percentile of the predicted outcomes from the PETS model. When 7 50, for instance, it measures the mean of the worst 50 percentile of predicted rewards. This is a pessimistic evaluation of the prospects of the action sequence A - it only pays attention to the worst performing particles in the distribution. At caution = 0, R., exactly matches the definition of in equation]] [ it measures the mean predicted reward of all the particles produced after particle propagation through the model. In experiments, we heuristically y 50.
Now, define "y-tcuuiouus action policy" action score R.,. In other words, A; = arg A R., (A). We propose to deploy such y-cautious action policies at adaptation time.
that selects actions based the generalized
4

---

Under review as a conference paper at ICLR 2020
The intuition is straightforward: in an unknown environment, the agent deliberately performs cautious actions to perform safe adaptation. Even though it eventually seeks to achieve high mean performance in the target environment, it does not select actions that achieve the highest expected reward under its model. Instead, it chooses to be conservative, not trusting the model's expectations fully.
4.3 MODEL FINETUNING
As it gathers experience in the target environ- ment using the y-cautious policy, RADA also improves its dynamics model over time to fine- tune it to the new environment. Since dynam-
Algorithm 1 RADA
1:
procedure PRETRAINING
2:
Initialize the prohabilistic ensemble dynamics model
3:
Initialize data D using random controller in ramnoo training for for domain ID a training domains do
ics models do not need any manually specified
4:
reward function during training, the ensemble model can continue to be trained in the same way as during the pretraining phase.
5:
Train the probabilisic ensemble dynamics model f on D for t task botizon do for evolutionary search siage=112... de for sampled action sequence A do Run state propagation to produce N particles Evaluate as R () Σ, r//N
6:
7:
8:
We propose to stabilize adaptation by keeping the model close to the original model. To do this, we maintain a replay buffer of data from the pretraining episodes, conducted outside the tar- get domain. During adaptation, model updates on this full dataset, including the
9:
10:
for
12:
Refine search find A max R(AA)
13:
end for
14:
Execute first action of A
compute all
15:
Record outcome in D
16:
end
replay buffers. We use probabilistic neural net-
17: end for
18:
end procedure
work ensembles for the dynamics model Chua
19:
procedure ADAPTAIION(P.)
et al. (018). and training proceeds through stochastic gradient descent.
20:
for larget dormin adaptaation 1pisode==12..... do
21:
for t 0 task harizan do for evolutionary search stage=1.2 1.2.... do
22:
23:
for sampled action sequence
As the model improves over time, the distribu-
24:
Run state propagation Evaluate with generalized score R., (A)
tion of predicted outcomes becomes more and
25:
more narrow over time. For deterministic en- vironment, the model eventually converges to deterministic predictions, so that R. is the same
26:
27:
Refine scarch find A" max R, (A)
28:
end for
29:
Execute first action A Record outcome in D
for all y. In other words, once the model is well-
30:
trained, the y-ccutioous action policy is identical
31:
Finetune the prohahilistic ensemble model f on
32:
end
to the standard action policy. The adaptation pro-
33:
end for
cedure in Algorithm [ sums up cautious action selection and model finetuning.
34:
end procedure
EXPERIMENTS
We now evaluate various ablations of RADA in driving environment, evaluating the importance of three components of our technique for generalization to unseen out-of-domain environments: (i) pretraining on multiple domains (i.e. 'domain randomization'), (ii) replaying pretraining stabilize finetuning in the target environment, (iii) y-cautious action policy (y >> 0) at adaptation time we heuristically set 7 50 for our experiments. RADA the version that includes all three techniques. For all methods, we use a PETS ensemble of 5 fully connected models, with each one having 4 layers. We hypothesize that being cautious would not only allow RADA adapt quicker by avoiding risk of catastrophic failure, but also that as uncertainty about the environment is resolved during adaptation, a cautious policy will become similar to the optimal policy, leading to higher final reward.
Baselines.
Our first baseline is RADA without cautious adaptation in the test environ- ment: RADAticaution
Next we separately ablate out multi-domain pretraining and pre- training replay from this approach to create two more baselines: RADA) (caut ion, DR} and RADA) { (caution, replay). Through these baselines, we systematically compare the contribu- tions of various components of RADA. We also compare RADA against PETS trained directly in the target environment, train-on-targer
As an external baseline, compare against robust adversarial reinforcement learning (RARL) Pinto jet al. (2017). RARL works by training a the actions emitted by the agent. We train the adversary to perturb the motor torques in Duckietown.
model-free free RL agent jointly with an adversary that perturbs
5

---

Under review as a conference paper at ICLR 2020
STANT
Figure 1: (left) Overhead and (right) first-person view from the car camera, of the Duckietown- driving environment.
Finally, we also implemented three meta-learning approaches as baselines: GrBal (Nagabandi et al., 2018 RL (Duan et al. 2016), and MOLe (Nagabandi et al. 2018b). However, in our experiments, all three metalearning approaches experienced training failures in our environment, failing to consistently reach the goal even during the pretraining phase. Consequently, we do not consider these methods the more detailed results reported in this section.
Car driving environment. Our driving environment is based on Duckietown (Chevalier- Boisserrt (ct al. 2018). a physically accurate driving environment designed for sim-to-real transfer. The task, illustrated Fig T. is to make right turn around corner to reach fixed goal tile. Each tile is fixed to size of 0.585. We modify the environment such that when the car attempts to take an action that would drive it off of the road tiles, it is held in place. If the the agent has learned to reverse out from the corner and try the turn again. The task rewards faster, more efficient turns, so that there is incentive go near the corner. At the same time, since there is big price to pay for it, cautious agent must avoid hitting the corner at all costs. The agent can observe its current æ, y coordinates, current velocity, and steering angle. At each time step, it given reward equal to negative Manhattan distance from the goal, with completion bonus of 100 if it successfully reaches the goal tile. The agent has direct control of the steering angle and velocity of the car at each time step. See [
clips the corner, it gets stuck unless
Each domain is specified by -ie-iiensioonal domain ID, the width of the car, which unknown to the agent. During pretraining, the car width ranges from 0.050 to 0.099, and is sampled uniformly from this range before each training episode. Having driven these cars of varying widths, we evaluate each method's s ability to adapt to driving cars. We test adaptation to one in-distribution domain: width 0.075, and five dii---idiitiiuttion domains: 0.1, 0. 0.125, 0.15, 0.175, and 0.20. We vary the car width because of its direct influence the optimal trajectory of the wider must make wider turns to avoid catastrophe.
Performance Metries. We measure the return (sum of rewards over full episode) and the number of collisions underwent in the target environment. For each method, we report the "average maximum reward" over adaptation time which the average over 10 random seeds of the maximum t adaptation episodes of the reward obtained in the target environment. Finally, to measure the safety of the adaptation process, we also report the cumulative number of collisions suffered by each method during adaptation, which more directly measures the extent to which different methods avoid catastrophic failures and perform safe adaptation.
Results. For all RADA variants, we perform pretraining for 32 episodes: 2 initial rollouts with random policy and 30 on-policy episodes. RADA (caution, DR} is pretrained for the same 32 episodes but single training domain the one with car width 0.1, which is the closest to the out-of-domain target car widths (recall the training car widths are 0.050-0.099), RARL trains model-free policies. which typically require more pretraining episodes; we use the default settings and train for 1000 episodes in the pretraining environments.
Fig 2 shows the average maximum reward after 10 adaptation episodes and the average total number of collisions for each method, as a function of the car width. All methods perform worse farther away from the training car widths, but RADA maintains its performance up to width 0.15, and deteriorates more gracefully, maintaining good performance even up to car width 0.2, over two times the largest width observed pretraining time. Not only does RADA achieve the highest rewards,

---

Under review as a conference paper at ICLR 2020
Rewards vs Target Car Widths
Domain
Out-of-Doo ootooinDomain
Return
Width
Boundary Collisions vs Target Car Widths
60
40
In-Domain
Out Out-of-Domain
RADA
(cuutocC)))
MAAuticattion,replly
-20
RARL
train-on-target
Width
Figure Evaluation of the average maximum reward and average total boundary collisions different target car widths.
Maximum Reward
Car Width 0.075
Car Wieth 0.1
Car Width 0.125
-20
Car Width 0.15
Car Width 0.175
Car Width 0.2
-20
Misl/ccarla,.R))
-40
-40
12
Figure 3: Evaluation of the reward over time at six different car width domains. For every evaluation. we perform ten adaptation steps, with one initial evaluation before adaptation starts. Each calculated reward is caleulated by taking the maximum reward seen far at each timestep, where the value that timestep is averaged over ten evaluations. This maximum is then averaged over ten initializations for each model. Standard errors are shown with colored bars.

---

Under review as a conference paper at ICLR 2020
Average Total Number of Boundary Collisions
Car
Width
30
Non Boundery
30
Car Width 0.15
Car Width 0.175
Car Width 0.2
RADiicastion (MOtticaationoo DR3 (cmuttonn
20
Adaptition Eptsedey
Adapiiaion Eatandery
Adaplation Episedes
Figure 4: Evaluation the total number of boundary collisions over time at six different car width domains. The maximum number of collisions is 60, as there are evaluations steps (1 before adaptation starts, during) and each evaluation step 10 evaluations are performed. This cumulative total is averaged over ten initializations for each model. Standard errors shown with colored bars.
but it also achieves them while being the most safe - it suffers the least number of collisions during adaptation across all these environments.
Comparing RADA ablations, cautious action selection during adaptation proves critical to perfor- mance, and RADA cautiii does much worse than RADA throughout. Domain randomization and pretraining replay have relatively smaller impacts on the performance after ten adaptation episodes, but we will show that they impact training speed and stability.
Training directly on the target domain (train-ove - on t target), aside from naturally resulting in more collisions, does not result in the best performance even terms of rewards. We believe that this is because pretraining on the training car widths has an additional advantage: it sets up curriculum for training that makes easier for the agent solve the exploration problem learning how to make the right turn.
Finally, RARL policies are relatively robust to changes in car width, but perform uniformly worse than RADA: they yield lower reward and experience more collisions at all car widths.
Adaptation speed. We now plot the results over adaptation time in each target environment for both the average maximum reward and the running total boundary collisions to show adaptation speed in the target environments. Fig shows the average maximum reward over adaptation time for various methods, and various target car widths. We evaluate one in-domain car width (0.075) and at five out-of-domain widths 10.1, 0.125, 0.15, 0.175, and 0.2). Across all six, RADA yields the highest rewards at all times except for car width 0. Fig 4 shows similar plots for the average cumulative total boundary collisions over adaptation time. Once again, RADA outperforms all approaches at most domains, with the least overall collisions at domains except for car width 0. Further, as seen in these cases, RADA (caattiionn DR} adapts more slowly than other approaches, demonstrating the value of multi-domain pretraining. Further RADA \ (cautic i on, repl ay) leads to very unstable training, reflected here in the fact that maximum reward over adaptation time does not monotonically increase, unlike the other approaches.
Epistemic uncertainty. RADA relies on the fact that a probabilistic dynamics model trained on population of source domains with varying unknown values of the domain ID learns capture
8

---

Under review as a conference paper at ICLR 2020
Figure 5: How well does the pretrained dynamics model capture the epistemic uncertainty due unknown car width? These overhead views present the trajectories predicted by the dynamics model (thin curves in desaturated colors) for a given starting state of the and specific action sequence. Overlaid on top of these are ground truth trajectories of the car (thick curves) corresponding to varying car widths (legend in the rightmost panel). See text for details.
the epistemic uncertainty associated with z. Fig visualizes how well the dynamics model learns to capture the variation in car behavior due to the unknown car width in our Duckietown setting. For in given state, evaluate specific action sequence, and plot the trajectories predicted by the dynamics model on an overhead view of the environment. These are shown in thin, desaturated curves in the panels in Fig 5 As the panels show, these trajectories originate at the and spread out as they get farther away, representing the higher uncertainty after more actions have been performed.
On top of these trajectories predicted by the model, overlay thicker curves corresponding to ground truth trajectories of cars of various widths, executing the same action sequence. In each panel of Fig 5 these ground truth trajectories all lie largely within the support of the predicted trajectory distribution, indicating that the model captures the epistemic uncertainty due to varying car widths.
particularly illustrative case is in the left-most panel, where the aggressive action trajectory takes the car very close to the corner of the road. This leads to erratic behavior that causes the car to swerve into the side of the road after clipping the corner some cases, or proceed successfully towards the goal near the center of the road in other cases. The distribution produced by the model's predicted trajectories are similarly bimodal, correctly capturing these two kinds of behavior. Appendix B Fig 7 shows how these predictions evolve over the course of RADA pretraining. Overall, these visualizations demonstrate that the dynamics model successfully learns to/ represent the epistemic uncertainty due unknown car width.
DISCUSSION
We have proposed RADA, a new approach to model-based reinforcement learning for safe, quick adaptation of RL agents in new environments with unknown dynamics. RADA relies two key ideas: transferring knowledge from training in a variety of training environments, and using a maximin notion of risk averrivrso during action selection in the target environment. We show in physically accurate driving environment that RADA performs fast, safe adaptation to learn to drive cars around corners, when they up two times larger than any it has driven pretraining time.
REFERENCES
Vivek Borkar and Rahul Jain. Risk-constrained markov decision processes, In 49th IEEE Conference on Decision and Control (CDC), pp. 1669.26669 IEEE, 2010.
Maxime Chevalier-ooisvert, Florian Golemo, Yanjun Cao, Bhairav Mehta, and Liam Paull. Duckietown environments for openai gym. bttps://githut. com/duckietoo//
gym-duckietown 2018.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for evar optimization in mdps. In Advances in neural information processing systems, pp. 3509-3511 3517, 2014.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. In Advances in Neural Information Processing Systems, pp. 1522-1530, 2015.

---

Under review as a conference paper at ICLR 2020
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 47544444777 4765, 2018.
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Picter Abbeel. 212: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02779, 2016.
Mohammad Ghavamzadeh, Shie Mannor, Joclle Pineau, Aviv Tamar, et al. Bayesian reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 8(5-6):359-44) 2015.
Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with uncertainty regularization for driving in dense traffic. arXiv preprint arXiv:1901.027005, 2019.
Gregory Kahn, Adam Villaflor, Vitchyr Pong, Picter Abbeel, and Sergey Levine. Uncertainty-awer reinforcement learning for collision avoidance. arXiv preprint arXiv:1702.01182, 2017.
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 199-806,25 2010.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Picter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt dynamic, real-world environments through meta-reinforcement 1347, 80.
learning. arXiv preprint
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta- meta-learning: Continual adaptation for model-based rl. arXiv preprint arXiv:1812.07677 2018b.
X. B. Peng, M. Andrychowicz. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 1-8, May 2018, doi: 10. 10.100//CRA.2013 2.1............5055.......
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 2817-2826. JMLR. org, 2017.
Aravind Rajeswaran, Sarvject Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint drXhv:1610.01283, 2016.
Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21-42, 2000.
Fereshtch Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without a single real image. In RSS, 2017.
Steindór Sæmundsson, Katja Hofmann, and Marc Peter Deisenroth. Meta reinforcement learning with latent variable gaussian processes. arXiv preprint 18X3:08003.0751,, 2018
Malcolm Strens. A bayesian framework for reinforcement learning. In ICML, volume 2000, pp. 943-950, 2000.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the evar via sampling. Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23-30. IEBE, 2017.
10

---

Under review as a conference paper at ICLR 2020
Boundary Collisions Target Car Widths
Total Numn Boundary Collistinn
50
40-
mn-Domain
of--o--n
20
Rewards Target Car Widths
80
pretrain
RADA (2 protrain envs)
RADA (5 pretrain envs)
40
20
In- mn-eomain
Out-of-Domain
Return
-20
-80
Figure Evaluation of the average maximum reward and average total boundary collisions at different target car widths. (similar to Fig but with small, fixed number of pretraining environments.)
A THE EFFECT OF FINITE PRETRAINING ENVIRONMENTS
In the main paper, we perform experiments where we sample the car width dynamically at beginning of each pretraining episode, from the full pretraining range (width 0.05 to 0.099), that there are effectively an infinite number pretraining cnvironments available to sample from. A more realistic situation might involve having fixed, small number of pretraining environments, among which we select one at random for each pretraining episode.
We now report the results of preliminary experiments investigate the feasibility of RADA in this setting, For these experiments, we sample a small number (2, 5, or 10) of car widths from the original pretraining range and limit pretraining to those environments only.
Fig 6 shows these results the same format as Fig 2 There are significant gains performance (both reward well as number of collisions) from 2 to 5 to 10 pretraining environments. 10 pretraining environments, results approach those reported earlier in Fig 2
Overall, these results indicate that RADA is feasible when the number of pretraining environments is finite and small.
B EVOLUTION OF DYNAMICS MODEL PREDICTIONS DURING TRAINING
we plotted the trajectory predictions by the fully pretrained dynamics model for fixed starting state and action sequence. Here, we show how the dynamics model's 's trajectory predictions improve over pretraining time. To do this, we replot the leftmost panel in Fig S but this time using dynamics models from various stages of pretraining. Fig shows the results. At first, the predictions are compeletely random and the predicted trajectories occupy the entire state space. They become more accurate throughout the course of training, approaching the trajectories plotted in the leftmost
In Figure 5
11

---

Under review as a conference paper at ICLR 2020
Figure 7: How do the dynamics model's predicted trajectories evolve pretraining time? These plots demonstrate the predicted trajectories for a fixed action sequence at different stages of RADA pretraining, starting at the first episode of training after the initial random rollouts.
image of Fig 5 As the model trains in the pretraining environments, it gradually learns to model the epistemic uncertainty associated with the unknown car width.
12


Reviews:
[{"review_id": "HJltqtIeKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of safe adaptation to avoid catastrophic failure in a new environment. It draws intuition from human behavior. The proposed method (risk-averse domain adaptation (RADA)) learns probabilistic model-based RL agents from source domains, and uses them to select actions that has the best worst-case performance in the target domain.\n\nThe paper mentions safety-critical applications like auto-driving. However, generally, I don't think black-box models are suitable for these safety-critical applications.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BkePIn1g5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes to adapt RL agents from some set of training environments (which, in the current instantiation, vary in some simple respect) to a new domain. They build on a framework for model-based RL called PETS. \n\nThe approach goes as follows: \n\n2-step process\n * train probabilistic model-based RL agents in a “population of source domains”\n * dropped into new environment use “pessimistic exploration policy”\n\nThen at test time, in order to compute estimates for the rewards for each action the authors use a “particle propagation” technique for unrolling through their dynamics model .\n\nThe action is chosen by looking at the sum of the 0 through kth percentile rewards. \nThis is a weird choice. Why are they looking at a sum over quantiles vs a quantile itself?\n\nThe claim is that the models from the first stage capture the epistemic uncertainty due to not knowing z.\nHowever, the authors give a too scant a treatment of what these uncertainty estimates really mean.\nFor example, they appear to only be valid with respect to an assumed distribution over z.\nThe paper’s experiments however focus in large part on what happens when the model is evaluated \non values of z that were outside the support of the distribution over training domains. \nIn this case, any benefit appears to be ill explained by the underlying motivation.\n\n\nThe next step here is to finetune the model as data is collected on the new domain.\n\nAuthors propose heuristics for this finetuning that include\n1. Drawing experiences from the past experiences (under different domains) and \n2. “keeping the model close to the original model”, via some sort of regularization presumably.\n\n>>> \twhy isn’t the exact nature of how they “keep the model near the original model explained in the text?\n\tperhaps the authors mean that 1. and 2. are one and the same (1 as  means to achieve 2)\n\tif this is the case, then the exposition should be improved to make this more clear.\n\n\nSome important details appear to be missing. For example, how many distinct source domains are seen during pretraining? Do they set z different z for every single episode of pretraining? Some language here is unclear, for example what precisely does an “iteration” mean in the context of the experiments? \n\nThe choice to report “average maximum reward” seems strange if what the authors care about is avoiding risk. Can they explain/justify this choice or if not, present a much more comprehensive set of experimental results?\n\nThe figures tracking catastrophic failures vs performance resembles those in \n“Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear”  https://arxiv.org/abs/1611.01211\nThis raises some question about why they don’t if concerned with “catastrophic events” model them more explicitly. \nElse, if the return accurately captures all desiderata, why to we need to count the failures?\n\nIn short this is a simpzle empirical paper that makes use of heuristic uncertainty estimates, \nincluding in settings when the estimates have no validity. The writing is reasonably clear\nand the ideas are straightforward (which is perfectly fine!). A few of the decisions are unnatural,\na few are ad hoc, and a few details are missing. Overall my sense is that this paper \nhas some good qualitities, including the clarity of much of the exposition, \nbut it’s still below the mark to be an impactful ICLR paper. \n\n==========UPDATE=================\nI read the rebuttal and am glad that the authors took time to read my review and engage with the criticism as well as try to make some small improvements to the paper, especially exploring the impact on the number of training environments on the results (in the original paper the number of environments available at train time was unlimited). The answers to some of the other questions were less convincing. E.g. the seemingly incoherent objective of summing over the quantiles falls flat. Why should we care more about being a \"strict generalization\" of some previous algorithm built upon than of having a coherent objective? Overall, I don't think the paper makes it over the bar to accept but I hope the authors continue to improve upon the work and get it into shape where it could be accepted at another strong conference.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xrQKZ3Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to address the safe adaptation: given a model trained on a variety of past experiences for some task, train a model learning to perform that task in a new situation while avoiding catastrophic failure. \n\n\nPros:\n- The idea of training on data from varying quartiles, with the goal of preventing overly-conservative models, is quite intriguing and inspiring.\n\nCons & Question:\n- Motivation:  \nCautious exploration or optimizing the best worst-case performance is conflicting with the philosophy of exploration, such as UCB. As stated in the introduction, “enables fast yet safe adaptation within only a handful of episodes.” Intuitively, we can not expect to be safe and fast at the same time. It would be better to discuss why cautious exploration can ensure fast and safe adaption, which would be more interesting. Additionally, in Figure 3, some fast adaption methods, such as MAML, should be compared to be more persuasive.\n\n- Method:\n 1. In equation (1), sum_N —> sum_i.\n 2. This work formulated safe adaption as minimizing the risk of catastrophic failure. What’s the relationship between “the generalized action score” and “risk of catastrophic failure”? The “generalized action score” is the main difference with PETs. However, it is a little bit hard to follow the idea from “risk of catastrophic failure” to “the generalized action scores”. \n 3. “Model-based RL agents contain dynamics models that can be trained in the absence of any rewards or supervision.” \n ”Since dynamics models do not need any manually specified reward function during training, the ensemble model can continue to be trained in the same way as during the pretraining phase.” I am confused about these sentences. Without the reward, what’s the purpose of RL? \n\n\n- Experiments:\n1. As stated in experiments, three meta-learning approaches have been deployed as baselines, including GrBal, RL^2 and MOLe. However, the experimental results are missing. Why meta-learning baselines do not work? Are there any explanations?\n2. There are many robust RL baselines, such as \n [1] Pinto L, Davidson J, Sukthankar R, et al. Robust adversarial reinforcement learning[C]//Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017: 2817-2826. \nIt would be better to compare with robust reinforcement learning work since there are no other baselines apart from meta-learning methods.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work this paper presents is interesting, but it is not quite ready yet for publication at ICLR. Specifically, the motivation of particular choices could be better, such as summing over quantiles, as indicated by Reviewer 1. The inherent trade-off between safety and speed of adaptation and how this relates to the proposed method could also use a clearer exposition.
Decision: Reject