Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
Authors: Benjamin Eysenbach
Shreyas Chaudhari
Swapnil Asawa
Sergey Levine
Ruslan Salakhutdinov
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
transfer learning
domain adaptation

Abstract: We propose a simple, practical, and intuitive approach for domain adaptation in reinforcement learning. Our approach stems from the idea that the agent's experience in the source domain should look similar to its experience in the target domain. Building off of a probabilistic view of RL, we achieve this goal by compensating for the difference in dynamics by modifying the reward function. This modified reward function is simple to estimate by learning auxiliary classifiers that distinguish source-domain transitions from target-domain transitions. Intuitively, the agent is penalized for transitions that would indicate that the agent is interacting with the source domain, rather than the target domain. Formally, we prove that applying our method in the source domain is guaranteed to obtain a near-optimal policy for the target domain, provided that the source and target domains satisfy a lightweight assumption. Our approach is applicable to domains with continuous states and actions and does not require learning an explicit model of the dynamics. On discrete and continuous control tasks, we illustrate the mechanics of our approach and demonstrate its scalability to high-dimensional~tasks.

-FF-DYNAM REINFORCEMENT LEARNING:
TRAINING FOR TRANSFER WITH DOMAIN CLASSIFIESS
Benjamin Eysenbach"
Shreyas Chaudhari CMU
Swapnil Asawa" University of Pittsburgh
CMU, Google Brain beysenbalcss cmu edu
shreyaschaudhariee cmu. edu
swa120pies edu
Sergey Levine UC Berkeley, Google Brain
Ruslan Salakhutinov CMU
ABSTRACT
We propose a simple, practical, and intuitive approach for domain adaptation reinforcement learning. Our approach stems from the idea that the agent's ex- perience in the source domain should look similar to its experience in the target domain. Building off of probabilistic view of RL, we achieve this goal by com- pensating for the difference in dynamics by modifying the reward function. This modified reward function is simple estimate by learning auxiliary classifiers that distinguish source-donas the agent is penalized for transitions that would indicate that the agent is interact- ing with the source domain, rather than the target domain. Formally, we prove that applying our method in the source domain is guaranteed to obtain a near-optimal policy for the target domain, provided that the source and target domains satisfy a lightweight assumption. Our approach applicable to domains with continuous states and actions and does not require learning explicit model of the dynam- ics. On discrete and continuous control tasks, we illustrate the mechanies of our approach and demonstrate its scalability to high-dimensional tasks.
transitions from target-domain transitions. Intuitively,
INTRODUCTION
Reinforcement learning (RL) can automate the acquisition of complex behavioral policies through real-world trial-and-error experimentation. However, many domains where we would like to learn policies are not amenable to such trial-and-error learning, because the errors are too costly: from autonomous driving to flying airplanes to devising medical treatment plans, safety-critical RL prob- lems necessitate some type of transfer learning, where a safer source domain, such as a simulator, is used to train policy that can then function effectively in target domain. In this paper, we examine a specific transfer learning scenario that we call domain adaptation, by analogy to domain adaptation problems in computer vision (Csurka 2017), where the training process in source domain can be modified so that the resulting policy is effective in given target domain.
RL algorithms today require large amount of experience the target domain. However, for
Real
Simulator (or Learned Model)
Simulator Learned Reward
many tasks may have access different but structurally similar source domain. While the source domain has different dynamics than the target domain, experience in the source do- main much cheaper collect. However, transferring policies from one domain to an- other challenging because strategies which effective in the source domain may not be effective the target domain. For example, ag- gressive driving may work well dry race- track but fail catastrophically on an icy road.
goal
start
goal
start
F(s.a) r(s,e) Ar(s.a)
classifier
Figure Our method acquires a policy for tar- get domain by practicing in the source domain using a (learedd) modified reward function.
"Equal contribution.

---

While prior work has studied the domain adaptation of observations in RL (Bousmalis et al. 2018
Ganin et al. 2016 Higgins et al. 2017). it ignores the domain adaptation of the dynamics.
This paper presents simple approach for domain adaptation in RL, illustrated in Fig. Our main idea is that the agent's experience in the source domain should look similar to its experience in the target domain. Building off of probabilistic view of RL., formally show that goal by compensating for the difference in dynamics by modifying the reward function. This mod- ified reward function is simple estimate by learning auxiliary classifiers that distinguish source- domain transitions from target-domain transitions. Because our method learns a classifier, rather than a dynamics model, we expect it to handle high dighesiieniio tasks better than model-based methods, a conjecture supported by experiments on the 1 11l-dmmnnioonll Ant task. Unlike prior work based on similar intuition (Koos er al., 2012, Wulfmeier et al. (017b), a key contribution of our work is formal guarantee that our method yields near-optimal policy for the target domain.
achieve this
The main contribution of this work is an algorithm for domain adaptation to dynamics changes in RL, based on the idea of compensating for differences in dynamics by modifying the reward function. We call this algorithm Domain Adaptation with Rewards from Classifiers, or DARC for short. DARC does not estimate transition probabilities, but rather modifies the reward function using a pair of classifiers. We formally analyze the conditions under which our method produces near- optimal policies for the target domain. On a range of discrete and continuous control tasks, we both illustrate the mechanics of our approach and demonstrate its scalability to higher-dimensional tasks.
RELATED WORK
While our work will focus on domain adaptation applied to RL, we start by reviewing more general ideas in domain adaptation, and defer to Kouw & Loog (2019) for recent review of the field. Two common approaches to domain adaptation are importance weighting and aomain-agnostic features. methods (c.g.. (Zadrozny 2004] Cortes & Mohri 2014 Lipton et al. 2018) estimate the likelihood ratio of examples under the target domain versus the source domain, and use this ratio to re- re-weight examples sampled from the source domain. Similar to prior work on importance weighting (Bickel et al.) 2007 Sønderby et al. 2016 Mohamed & Lakshminarayanan) 2016 Uehara et al.| 2016). our method will use a classifier to estimate probability ratio. Since we will need to estimate the density ratio of conditional distributions (transition probabilities), we will learn two classifiers. Importantly, we will use the logarithm of the density ratio to modify the reward function instead of weighting samples by the density ratio, which is often numerically unstable (see, e.g., .gg..Schulman et al. 2017 {3)) and led to poor performance in our experiments.
Prior methods for applying domain adaptation to RL include approaches based on system iden- tification, domain randomization, and observation adaptation. Perhaps the most established ap- proach, system identification (Ljung 1999), uses observed data to tune the parameters of simu- lator (Feldbaum) 1960 Werboss) 1989; Wittenmark] 1995] Ross & Bagnell, 2012 Tan et al., 2016 Zhu et al., 2017bb] Farchy et al. 2013) More recent work has successfully used this strategy to bridge the sim2real gap (Chebotar et al. 2019 Rajeswaran et al.| (0166). Closely related is work on online system identification and meta-learning which directly uses the inferred system param- eters to update the policy (Yu et al., 2017 CClavera et al.] 2018 Tanaskovic et al.) 2013 Sastry & Isidori 1989). However, these approaches typically require either model of the environment or manually-specinge distribution over potential test-time dynamics, requirements that our method will lift. Another approach, domain randomization, randomly samples the parameters of the source do- main and then finds the best policy for this randomized environment (Sadeghi & Levine 2016 Tobin al. 2017 Peng et al. 2018 Cutler et al., 2014)). While often effective, this method is sensitive to the choice of which parameters are randomized, and the distributions from which these simulator third approach, observation adaptation, modifies the observations of the source domain to appear similar to those in the target domain (Fernando et al., 2013 Hoffman ctt al., 2016; Wulfmeier et al. 2017a). While this approach has been successfully applied to video games (Gamrian & Goldberg 2018) and robot manipulation (Bousmalis et al. 2018). ignores the fact that the source and target domains may have differing dynamics.
parameters are sampled.
Finally, work is similar to prior work on transfer learning (Taylor & Stone 2009) and meta- learning in RL, but makes less strict assumptions than most prior work. For example, most work on meta-RI. (Killian et al. 2017 Duan et al. 2016 Mishra al. 2017: Rakelly et al. 2019) and
2

---

some work on transfer learning Perrinn al. 1999 Tanaka & Yamamura 2003 Summola & Wyatt 2006) assume that the agent has access to many source tasks, all drawn from the same distribution as the target task. [Selfridge al. ( (1985); Madden & Howley (2004) assume curriculum of tasks, [Ravindran & Barto (2004) assume that the source and target domains have the same dynamies locally, and [Sherstov & Stone (2005) assume that the set of actions that useful in the source domain is the same as the set of actions that will be useful in the target do- main. Our method does not require these assumptions, allowing it to successfully learn in settings where these prior works would fail. For example, the assumption of Sherstov & Stone (2005) is violated in our experiments with broken robots: actions which move a joint are useful in the source domain (where the robot is fully-function) but not useful in the target domain (where that joint is disabled). Our method will significantly outperform an importance weighting baseline (Lazaric) 2008). Unlike Vemula et al. (2020), our method does not require learning a dynamics model and is applicable to stochastic environments and those with continuous states and actions. Our algorithm bears resemblance to that in Wulfmeier et al. (2017b. but crucial algorithmic difference allows us to prove that our method acquires a near-optimal policy in the target domain, and also leads to improved performance empirically.
saauully-sppccffie
The theoretical derivation of our method is inspired by prior work which formulates control as a problem of probabilistic inference (e.g.. [oossaint] 2009 Rawlik al. 2013 Levine al. 2018))
Algorithms for model-based RL (e.g.. (Deisenroth & Rasmussen [201] Hafner et al. [018:] Jan- ner et al., 2 and off-policy RL (e.g.. (Munos et al. [016 Fujimoto al.] [0118 Dann et al. 2014 Dudik et al. 2011) similarly aim to improve the sample efficiency of RL, but do use the source domain accelerate learning. Our method is applicable any maximum entropy RL algo- rithm, including on-policy (Song et al. 2001)). off-policy (Abdolmaleki et al. 20188) Haarnoja et al.] 2018). and model-based (Janner et al.; [2019| WWilliass et al., (015) algorithms. We will use the SAC (Haarnoja et al., 2018) in our experiments and compare against model-based baselines.
3 PRELIMINARIES
In this section, we introduce notation and formally define domain adaptation for RL. Our problem setting will consider two MDPs: Masurce represents the source domain (e.g.. a practice facility, simulator, or learned approximate model of the target domain) while Munget represents a the target domain. We assume that the two domains have the same state space S, action space A, reward function and initially state distribution Pi (81); the only difference between the domains is the dynamics, Psource(5t41 St; de) and Pranger(St-4 [st+1 St, at). We will learn a Markovian policy aa s), parametrized by 0. Our objective is to learn a policy π that maximizes the expected discounted sum of rewards on Marrget Mix.Maps E d.)]. We now formally define our problem setting:
Definition 1. Domain Adaptation for RL is the problem of using interactions in the source MDP Mooure together with a small number of interactions in the target MDP Marrgr to acquire a policy that achieves high reward in the target MDP, Manrgr
We will assume every transition with non-zero probability in the target domain will have non-zero probability in the source domain:
Ptarger (8 8t, 0x) > 0 = Psource (8t-1 8t, az) > 0 for all 8t, 8t-1 E E at E A.
(1)
This assumption is common in work importance sampling (Koller& & Friedman, 2009 (12.2.2),
and the converse need not hold: transitions that are possible in the source domain need not be possible in the target domain. If this assumption did not hold, then the optimal policy for the target domain might involve behaviors that are not possible in the source domain, so it is unclear how one could learn near-optimal policy by practicing in the source domain.
4 A VARIATIONAL. PERSPECTIVE ON DOMAIN ADAPTATION IN RL
The probabilistic inference interpretation of RL Kappen 2005 Todorov 2007 Toussaint 2009
Ziebart 2010 Rawlik et al. 2013 Levine 2018) treats the reward function as defining a desired distribution over trajectories. The agent's 's task is to sample from this distribution by picking trajec- tories with probability proportional to their exponentiated reward. This section will reinterpret this model in the context of domain transfer, showing that domain adaptation of dynamics can be done
3

---

by modifying the rewards. To apply this model to domain adaptation, define p(7) the desired distribution over trajectories in the target domain,
p(T) Pi (81)
(I Pranger (8
se; ar) exp
Σ Er(ssaa)))
and ((T) as our agent's distribution over trajectories in the source domain,
(()) (11)
II Psource (S 8t, (y) st).
As noted in Section we assume both trajectory distributions have the same initial state distribution. Our aim is to learn a policy whose behavior in the source domain both receives high reward and has high likelihood under the target domain dynamics. We codify this objective by minimizing the reverse KL divergence between these two distributions:
min (x (a p) -PEpa Σ Er(sc,au) ai) + H. la sx] Ar(s s)) tc, =(a/8)
(2)
where
Ar(s++1, 8r, ax) A logp($141 5e, ae) log g(5++1 (8,
The constant e the partition function p(r), which is independent the policy and dynamics. While Ar is defined in terms of transition probabilities, in Sec. we show how to estimate Ar by learning a classifier. We therefore call our method domain adaptation with rewards from classifiers (DARC), and will DARC refer the policy that maximizes the objective in Eq.
Where the source and target dynamics are equal, the correction term Ar is zero and we recover maximum entropy RL (Ziebart 2010: Todorov (007). The reward correction different from prior work that adds log 3(a to the reward to regularize the policy to be close to the behavior policy | (e.g..Jaques et al. (217)) Abdolmaleki et al. 2018)). ). In the case where the source dynamics are nor equal to the true dynamics, this objective is not the same maximum entropy RL on trajectories sampled from the source domain. Instead, this objective suggests corrective term Ar that should be added to the reward function to account for the discrepancy between the source and target dynamics. The correction term, Ar, is quite intuitive. If a transition (st, at, $1+1) has equal probability in the source and target domains, then Ar(s;, at) are likely in the source but are unlikely in the target domain, Ar < O, the agent is penalized for "exploiting" inaccuracies or discrepancies in the source domain by taking these transitions. For the example environment in Figure [ transitions through the center the environment are blocked in the target domain but not in the source domain. For these transitions, Ar would serve large penalty, discouraging the agent from taking these transitions and instead learning to navigate around the wall. Appendix A presents additional interpretations of Ar in terms of coding theory, mutual information, and a constraint on the discrepancy between the source and target dynamics. Appendix[ddiseusee how prior work on domain agnostic feature learning can be viewed a special case of our framework.
0 so no correction is applied. For transitions that
4.1 THEORETICAL GUARANTEES
We now analyze when maximizing the modified reward r + Ar in the source domain yields a near- optimal policy for the target domain. Our proof relies the following lightweight assumption: Assumption 1. Let n arg max, Ep r(st, at) be the reward- maximizza policy in the target domain. Then the expected reward in the source and target domains differs by most 2.Raaxx ee/2:
E Bps- Er(sena))
EE PPurpet
Σ E((tt,a)) ai)
< 2.Rmaxx Ve/2.
The variable Rmax refers to the maximum entropy reguaaz-e return of any trajectory. This assump- tion says that the optimal policy in the target domain is still good policy for the source domain, and its expected reward is similar in both domains. We do not require that the opposite be true: the optimal policy in the source domain does not need to receive high reward in the target domain. If there are multiple optimal policies, we only require that this assumption hold for one them. We now state main result:
4

---

Algorithm Domain Adaptation with Rewards from Classifiers [DARC]
1: Input: source MDP Msorce and target Manger ratio r of experience from source vs. target.
2: Initialize: replay buffers for source and target transitions, Doourees Durgert policy n; parameters 0= (OSAS: 0sA) for classifiers Uosss (target 8t, de, 81+1) and yosss (target 8t, (t).
3: for = 1, .... num iterations do
4:
Dsource < sourre U ROLLOUT(T, (T,,OUUTTTTMMUCCC, Msource)
Collect source data. > Periodically, collect target data.
5:
i mod r = then
6:
Dunget <- Durget U RROLLOUT Mangel) <- 0 - v)((((
7:
Update both classifiers. Ar is computed with Eq.
8:
F(st, 0e; Se+1) r(st,aa) 04) + Ar(st, (t, (2, 8(+1)
9:
+ MAXENT [L(T,, Dsouree,
10: return
Theorem 4.1. Let DARC be the policy that maximizes the modified (entropy-reuularize) reward in the source domain, let n be the policy that maximizes the (unmodified, mutropy-regulare reward in the target domain, and assume that πt" satisfies Assumption
[
Then the following holds:
Σ a() + H(ae [ss
Σ (((s + H(aa | sil] 4mmmx v e/2.
E Proper 7f DARC
See Appendix for the proof and definition of This result says that DARC attains near-optimal centropy-regularizer reward on the target domain. Thus, we can expect that modifying the reward function should allow us to adapt to different dynamics. The next section will present practical algorithm for acquiring DARC by estimating and effectively maximizing the modified reward in the source domain.
DOMAIN ADAPTATION IN RL WITH A LEARNED REWARD
The variational perspective on model-based RL in the previous section suggests that we should modify the reward the source domain by adding Ar. In this section we develop practi- cal algorithm for off-dynamics RL by showing how Ar be estimated without learning explicit dynamics model.
Source
Eaperience Source Domsin
(Jinited) Experience
Reward Commttion,
(r(s,
Agent
Two Clasilters
2. Updatt classifiers to distinguish target transitions.
Train agent with modified rewerds.
To estimate Ar, will use pair of (learedd) binary classifiers, which will infer whether transitions came from the source or target do- main. The key idea is that the transition proba- bilities are related the classifier probabilities via Bayes' rule:
Collect experience from source and (limited) target.
Figure 2: Block diagram of DARC (Alg.
p(target at 8141)
(( St, Ot, target) p(st, a, (arget))(target)/lls.. 81+1).
= 19
We estimate the term p(s,, dr target) on the RHS via another classifier, p(tagget 8t, a)):
p(target
St, (t,aa))(ss, at)
p(st, ag target)
pttarget)
Substituting these expression into definition for Ar and simplifying, obtain estimate for Ar that depends solely on the predictions of these two classifiers:
Ar(Se, Ge: 8t+1)
log p ptargeet St: 8141) log ptargeet 8t, at)
p(surcee | Se, Cey St41) log p(source |t, st,(a)) az)
(3)
The orange terms are the difference logits from the classifier conditioned on 8t, a,, S141, while the blue terms are the difference in logits from the classifier conditioned just 8t, at Intuitively,
5

---

Ar answers the following question: for the task of predicting whether transition came from the source or target domain, how much better can you perform after observing 8t-1? ? We make this connection precise in Appendix A.2 by relating Ar mutual information. Ablation experiments (Fig. 7 confirm that both classifiers are important to the success of our method. The use of transition classifiers makes our method look somewhat similar adversarial imitation learning (Ho & Ermon 2016 Fu et al., 2017). While our method is not solving an imitation learning problem (we do not assume access to any expert experience), our method can be interpreted as learning a policy such that the dynamics observed by that policy in the source domain imitate the dynamics of the target domain.
Algorithm Summary Our algorithm modifies an existing MaxEnt RL algorithm additionally learn two classifiers, 90ass (target St, aq, 8t+1) and 90şa (target St, az), parametrized by ASAS and ASA respectively, to minimize the standard cross-entropy loss.
2 -Duun [log 98 (target Se: Ot, St+1)] Dx [log 98s, (source St, St+1)l
lSA (OSA) A EDDins [log 90sa (target St,, a:)] ED... [log 90a, (source St, ac)]
Our algorithm, Domain Adaptation with Rewards from Classifiers (DARC), is presented in Alg. and illustrated in Fig. [
To simplify notation, we define 0 _ (OSAS, OSA) and {(0) - EsAs + fsA (0sa). At each iteration, we collect transitions from the source and (less frequently) target do- main, storing the transitions in separate replay buffers. We then sample batch of experience from both buffers to update the classifiers. We use the classifiers to modify the rewards from the source domain, and apply MaxEnt RL to this experience. We use SAC (Haarnoja et al. 2018) MaxEnt RL algorithm, but emphasize that DARC is applicable to any MaxEnt RL algorithm (e.g., on-policy, off-policy, and model- When training the classifiers, add Gaussian input noise to prevent overfitting to the small number of target-domain transitions (see Fig. f an ablation).
6 EXPERIMENTS
We start with didactic experiment to build intuition for the mechanics of method, and then evaluate on more complex tasks. Our experiments will show that DARC outperforms alternative ap- proaches, such as directly applying RL to the target domain or learning importance weights. We will also show that our method can account for domain shift in the termination condition, and confirm the importance of learning two classifiers.
Illustrative example. We start with a simple gridworld example, shown on the right, where we can apply our method without function ap- proximation. The goal to navigate from the top left to the bottom left. The real environment contains an obstacle (shown in red), which is not present in the source domain. If we sim- ply apply RL on the source domain, we obtain policy that navigates directly to the goal (blue arrows), and will fail when used in the target do- main. We then apply our method: we collect trajectories from the source domain and real world to fit the tabular classifiers. These classifiers give modified reward, which we use learn policy in the source domain. The modified reward causes our learned policy to navigate around the obstacle, which succeeds in the target environment.
Goal
RL on Source Domai
DARC (Ouns)
Donaan
Figure 3: Tabular example of off-dynamics RL
Visualizing the reward modification in stochastic domains. In our next experiment, we use an "archery" task to visualize how the modified reward accounts for differences in dy- namics. The task, shown in Fig. 4 requires choosing an angle at which to shoot an arrow. The practice range (i.e., the source domain) is outdoors, with wind that usually blows from left to right. The competition range (i.e., the target domain) is indoors with no wind. The re- ward is the negative distance the target. We
(8l)
reward on sou reaend on target mosifted reward
simulaner
world
200
Figure 4: Visualizing the modified reward

---

Reward (Target)
broken reacher
let broken half cheetah
2+3
broken
half cheetah obstacle
1.53
-22
0.75
-23
0.4
13
1.3
Eav. Staps (Target)
345
Steps (Target)
Env. Steps (Target)
Env. Steps (Target)
DARC (ours)
Target (1e6 stops)
Finetuning
Importance Weighting
PETS
RL on Target
RL en Source
RL on Target (10x steps)
MBPO
MATL
Figure 6: DARC compensates for crippled robots and obstacles: We apply DARC to four continuous control tasks: three tasks (broken reacher, half cheetah, and ant) which are crippled in the target domain but not the source domain, and one task (half cheetah obstacle) where the source domain omits the obstacle from the target domain. Note that naïvely ignoring the shift dynamics (green dashed line) performs quite poorly, while directly learning on the crippled robot requires order of magnitude more experience than method.
plot the reward as a function of the angle in both domains in Fig. 4 The optimal strategy for the outdoor range is to compensate for the wind by shooting slightly to the left (0 -0.8, while the optimal strategy for the indoor range is to shoot straight ahead (0 = 0). We estimate the modified reward function with DARC, and plot the modified reward in the windy outdoor range and indoor range. We aggregate across episodes using J(0) = log |o [exp(r(s))) see Appendix E.4 for details. We observe that maximizing the modified reward in the windy range does not yield highh reward in the windy range, but does yield a policy that performs well in the indoor range.
Scaling more complex tasks. We now ap- ply DARC to the more complex tasks shown in Fig.5 5
We define three tasks by crippling one of the joints of each robot in the target domain, but using the fully-funcoe fully-functional robot in the source domain. We use three simulated robots taken from OpenAl Gym (Brockman et al. 2016): 7 DOF reacher, half cheetah, and ant. The broken reacher is based the task described by Vemula eet al. (2020). We also include a task where the shift in dynamics is external to the robot, by modify- ing the cheetah task reward the agent for running both forward and backwards. It is easier to learn to run backwards, an obstacle in the target domain prevents the agent from running backwards. This "half cheetah obstacle" task does not entirely satisfy the assumption in Eq. [ because transitions such as bouncing off the obstacle are only possible in the target domain, not the source domain. Nonetheless, the success of our method on this task illustrates that DARC can excel even in settings that do not satisfy the assumption in Eq.
Figure 5: Environments: broken reacher, broken half cheetah, broken ant, and half cheetah obstacle.
We compare our method to eight baselines. RL on Source and RL on Target directly perform RL on the source and target domains, respectively. The Finetuning baseline takes the result of running RL on the source domain, and further finetunes the agent on the target domain. The Importance Weighting baseline performs RL importance--eiighed weighted samples from the source domain; the importance weights are (xp(Ar). Recall that DARC collects many (r = 10) transitions in the source domain and performs many gradient updates for each single transition collected in the target domain (Alg. [5.. We therefore compared against a RL on Target (10x) baseline that likewise performs many (r 10) gradient updates per transition in the target domain. Next, we compared against two recent model-based RL methods: MBPO (Janner et al. (2019) and PETS (Chua et al., 2018). Finally, we also compared against MATL (Wulfmeier al. 2017b), which is similar in spirit to our method but uses a different modified reward.
We show the results of this experiment in Fig. 6 plotting the reward on the target domain as a function of the number of transitions in the target domain. In this figure, the transparent lines correspond to different random seeds, and the darker lines are the average of these random seeds. On all tasks, the RL on source baseline (shown as a dashed line because it observes no target transitions) performs considerably worse than the optimal policy from RL on the target domain, suggesting that good policies for the source domain are suboptimal for the target domain. Nonetheless, on three of the four tasks our method matches (or even surpasses) the asymptotic performance of doing RL on the target domain, despite never doing RL experience from the target domain, and despite observing 5 10x less experience from the target domain. On the broken reacher and broken half cheetah tasks, finetuning the target domain performs par with our method. On the simpler
7

---

-20
Rewwrr (Target)
Reward (Target)
-40
-40
Myn MM
0=0.0
-60
-60
0=
-80
With SA Classifier (ours)
#=1.0
Without SA Classifier
10.0
-100
-100
0.2
0.4
0.6
1.0
0.4
0.8
Env. Steps (Target)
145
Env. Steps (Target)
Figure 7: Ablation experiments (Left) DARC performs worse when only one classifier is used. (Right)
Using input noise to regularize the classifiers boosts performance. Both plots show results for broken reacher, see Appendix Dioor results on environments.
broken reacher task. just doing RL on the target domain with large number of gradient steps works quite well (we did not tune this parameter for our method). While the model-based baselines (PETS and MBPO) also performed well on for low-dimensional tasks (broken reacher, broken half cheetah), they perform quite poorly on more challenging tasks like broken ant, supporting our intuition that classification is easier than learning dynamics model in high dimensional tasks. Finally, DARC outperforms MATL on all tasks.
Ablation Experiments. Our next experiment examines the importance of using two classifiers to estimate Ar. We compared our method to an ablation that does not learn the SA classifier, effectively ignoring the blue terms in Eq. As shown in Fig. (left), this ablation performs considerably worse than method. Intuitively, this makes sense: might predict that transition came from the source domain not because the next state had higher likelihood under the source dynamics, but rather because the state action visited more frequently in the source domain. The second classifier used in our method corrects for this distribution shift.
Next, examine the importance of input noise regularization in classifiers. As we observe only a handful of transitions from the target domain, we hypothesized that regularization would be im- portant to prevent overfitting. We test this hypothesis in Fig. (right) by training our method on the broken reacher environment with varying amounts of input noise. With no noise or little noise our method performs poorly (likely due overfitting); too much noise also performs poorly (likely due to underfitting). We used value of in all our experiments, and did not tune this value. See Appendix for more plots of both ablation experiments.
To gain more intuition for our method, we
Training
recorded the reward correction Ar throughout training on the broken reacher environment. In
Reward Correction (Ar)
without Ar
Training with Ar
this experiment, we ran RL on the source do- main for 100k steps before switching to our method. Said another way, we ignored Ar for the first 100k steps of training. As shown in Fig. B] Ar steadily decreases during these first 100k steps, suggesting that the agent learn- ing strategy that takes transitions where the source domain and target domain have different dynamics: the agent is making use of its broken joint. After 100k steps, when we maximize the combination of task reward and Ar, we observe that Ar increases, so the agent's transitions in the source domain are increasingly consistent with target domain dynamics. After around le6 training steps Ar is zero: the agent has learned a strategy that uses transitions that are indistinguish- able between the source and target domains.
Agent finds strategy the source and target domains are indistinguishable.
-6-
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Training Steps
166
Figure 8: Without the reward correction, the agent takes transitions where the source domain and target do- mains are dissimilar; after adding the reward correction, the agent's transitions in the source domain are increas- ingly likely under the target domain.
Safety emerges from domain adaptation to the termination condition. In many safety-critical applications, the real-world and simulator have different safeguards, which kick in to stop the agent and terminate the episode. For an agent to effectively transfer from the simulator to the real world, cannot rely on safeguards which are present in one domain but not the other. Since this termination condition part the dynamics (White 2017), 7), we can readily apply DARC to this setting.
8

---

Comparison with Prior Transfer Learning Methods. We are not the first work that mod- perform trans- We the humanoid shown in Fig. for this experiment and set the task reward to 0. In the source domain episodes have fixed length of 300 steps; in the target domain the episode terminates when the robot falls. The scenario mimics the real-world setting where robots have freedom to practice in a safe, cush- ioned, practice facility, but are preemptively stopped when they try to take unsafe actions in the real world. Our aim is for the agent to learn to avoid unsafe transitions in the source domain that would result in episode termination in the target domain. As shown in Fig. 2 our method learns to remain standing for nearly the entire episode. As expected. baselines that maximize the zero reward on the source and target domains fall immediately. While DARC was not designed
Time Before Falling
300
Time Liii
200
100
DARC (ours) AL on Target on
0-
0.5
Training Steps
Figure 9: Our method accounts for domain shift in the termination condition, causing the agent to avoid tran- sitions that cause termination in the target domain.
method for safe RL. (Tamar et al.) 2013) Accham et al., [017]] Eysenbach et al.] 2017]] Berkenkamp et al., 2017), this experiment suggests that safety may emerge automatically from DARC, without any manual reward function design.
1.0
ifies the reward function fer in RL (Koos et al., 2012). nor the first work learn how to modify the reward func- tion (Wulfmeier et al.) (2017a). However, these prior works lack theoretical justification. In contrast, our approach maximizes well- defined variational objective and our analysis guarantees that agents learned with method will achieve similar rewards in the source and target domains. Our formal guarantees (Sec. 4 do not apply to MATL (Wulfmeier et al.) 2017b) because their classifier is not conditioned on the action. Indeed, our results on the four tasks in Fig. indicate that DARC ourperforms MATL all tasks. To highlight this difference, we compared DARC and MATL on a gridworld (right), where the source and target domains differed by assigning opposite effects the "up" and "down" in the purple state in the source and target domains. We collected data from uniform random policy, so the marginal distribution p(s 8t) was the same in the source and target domains, even though the dynamics p(#++1 St, az) where different. In this domain, MATI fails recognize that the source and target domains are different. DARC succeeds in this task for 80% of trials while MATL. succeeds for 0% of trials. We conclude that the conditioning the action, suggested by our analysis, especially important when using experience collected from stochastic policies.
Success Rate
0.8
0.6
0.2
0.0
DARC
MATL
Figure 10: Comparison with MATL
DISCUSSION
In this paper, we proposed a simple, practical, and intuitive approach for domain adaptation to changing dynamics in RL. We motivate this method from novel variational perspective domain adaptation in RL, which suggests that we can compensate for differences in dynamics via the reward function. Moreover, guaranteed to yield a near-optimal policy for the target domain. Experiments on a range of control tasks show that our method can leverage the source domain to learn policies that will work well in the target domain, despite observing only a handful of transitions from the target domain.
formally prove that, subject lightweight assumption, our method is
The main limitation of our method is that the source dynamics must be sufficiently stochastic, an assumption that can usually be satisfied by adding noise to the dynamics, or ensem- bling collection of sources. Empirically, we found that our method worked best on tasks that could be completed in many ways in the source domain, but some these strategies were not compatible with the target dynamics. The main takeaway of this work is that inaceuracies in dynamics can be compensated for via the reward function. In future work we aim to use the variation perspective on domain adaptation (Sec. to learn the dynamics for the source domain.
Limitations

---

Acknowledgements. We thank Anirudh Vemula for early discussions; we thank Karol Hausman, Vincent Vanhoucke and anonymous reviews for feedback on drafts of this work. We thank Barry Moore for providing containers with MuJoCo and Dr. Paul Munro granting access to compute at CRC. This work is supported by the Fannic and John Hertz Foundation, University of Pittsburgh Center for Research Computing (CRC), NSF (DGE1745016, I1S1763562), ONR (N00014181 12261), and US Army. Any opinions, findings and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Contributions. BE proposed the idea of using rewards to correct for dynamics, designed and ran many of the experiments in the paper, and wrote much of the paper. SA did the initial literature review, wrote and designed some of the DARC experiments and environments, developed visualizations of the modified reward function, and ran the MBPO experiments. SC designed some the initial environments, helped with the implementation of DARC, and ran the PETS experiments. RS and SL provided guidance throughout the project, and contributed to the structure and writing of the paper.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Ried- miller. Maximum posteriori policy optimisation. arXiv preprint 1rXiv:1806.06922 2018.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning- Volume 70. pp. 22 31. JMLR. org, 2017.
Felix Berkenkamp, Mattco Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforeement learning with stability guarantees. Advances in neural information processing systems, pp. 908-918, 2017.
Steffen Bickel, Michael Brtickner, and Tobias Scheffer. Discriminative learning for differing training and test distributions. In Proceedings of the 24th international conference on Machine learning, pp. 81-88, 2007.
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, et al. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In 2018 IEEE International Conference on Robotics and Automation (ICRA). pp. 22434250. IEEE, 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint /60: 54500, 2016.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8973 8979. IEEE, 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754-4765, 2018.
Ignasi Clavera, Anusha Nagabandi, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learn- ing to adapt: Meta-leaming for model- control. arXiv preprint arXiv:1803.1134,. 3, 2018.
Corinna Cortes and Mchryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103-126. 2014.
Gabriela Csurka. Domain adaptation for visual applications: A comprehensive survey. arXiv preprint
arXiv:1702.05374, 2017.
Mark Cutler, Thomas Walsh, and Jonathan How. Reinforcement learning with multi-fidelity simulators. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 3888-3895. IEEE, 2014.
Christoph Dann, Gerhard Neumann, Jan Peters, et al. Policy evaluation with temporal differences: A survey and comparison. Journal of Machine Learning Research, 15:809 0803, 2014.
Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference machine learning (ICML-II, pp. (65-472, 2011.
Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett, Tlya Sutskever, and Pieter Abheel. R12: Fast reinforce- ment learning via slow reinforcement learning. arXiv preprint arXiv: 161 I. I.02799, 2016.
10

---

Miroslav Dudik, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv:1103.4611, 2011.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave trace: Learning reset for safe and autonomous reinforcement learning. arXiv preprint 1rXiv:171..0782, 2017.
Alon Farchy, Samuel Barrett, Patrick MacAlpine, and Peter Stone. Humanoid robots learning walk faster: From the real world simulation and back. In Proceedings of the 2013 international conference on Au- tonomous agents and multi-agent systems, pp. 39 46, 2013.
AA Feldbaum. Dual control theory. Avtomatika Telemekhanika, 21(9):120--1249 1960.
Basura Fernando, Amaury Habrard, Mare Sebban, and Tinne Tuytelaars. Unsupervised visual domain adapta- tion using subspace alignment. In Proceedings of the IEEE international conference on computer vision, pp. 2960-2967, 2013.
Justin Fu, Katie Luo, Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv: 1244,, 2017.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv:1812.02900 2018.
Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to- image translation. arXiv preprint arXiv: 2018.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Muchine Learning Reseurch, 17()):2096-2030, 2016.
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Chris Harris, Vincent Vanhoucke, et al. Tf-agents: A library for reinforcement learning in tensorflow, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv: 2018
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lec, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 1,0551,, 2018
Trina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforce- ment learning. Proceedings of 34th International Conference on Machine Learning- Volume 70, pp. 1480-1490. JMLR. org, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances neural information processing systems, pp. 4565-4573, 2016.
Judy Hoffman, Dequan Wang, Fisher Yu, and Trevor Darrell. Fens in the wild: Pixel-level adversarial and constraint-based adaptation. arXiv preprini urXiv:1612.02649, 2016.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pp. 12498-12509, 2019.
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hemánndzz-Loban Richard Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1645-1654, JMLR. org, 2017.
Hilbert J Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of statistical mechanics: theory and experiment, 2005(11):P11011, 2005.
Taylor W Killian, Samuel Daulton, George Konidaris, and Finale Deshi-Velez. Robust and efficient transfer learning with hidden parameter markov decision processes. In Advances in neural information processing systems, pp. 6250-6261, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6600 2014.
arXiv preprint
Daphne Koller and Nir Friedman. Probabilistcc graphical models: principles and techniques. MIT press, 2009.
11

---

Sylvain Koos, Jean-Baptiste Mouret, and Stéphane Doncieux. The transferability approach: Crossing the reality gap evolutionary robotics. IEEE Transactions on Evolutionary Computation, 17(1):122 115h1122245,,22 2012.
Wouter Marco Kouw and Marco Loog. A review of domain adaptation without target labels. JEEE transactions on pattern analysis and machine intelligence, 2019.
Alessandro Lazaric. Knowledge transfer in reinforcement learning. PhD thesis, PhD thesis, Politecnico di Milano, 2008.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXi:: 00000090, 2018.
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordi- nation rohotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5):4214433 2018.
Zachary C Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and correcting for label shift with black box predictors. arXiv preprint 1rxxiv1802.03916 2018.
Lennart Ljung. System identification. Wiley encyclopedia of electrical and electronics engineering, pp. 1-19, 1999.
Michael G Madden and Tom Howley. Transfer of experience between reinforcement learning environments with progressive difficulty. Artificial Intelligence Review, 21(3-4):375-398, 2004.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49(2-3):267- 290, 2002.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive learee. arXiv preprint arXiv: 77X1:1770703314,,201
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv: arXiv: 1610.03433, 2016.
arXiv preprint
Rémi Munos, Tom Stepleton. Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforce- ment learning. In Advances Neural Information Processing Systems. pp. 1054-1062, 2016.
Xue Bin Peng, Marcin Andrychowicz. Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference robotics and automation (ICRA), pp. 1-8. JEEE, 2018.
Theodore J Perkins. Doina Precup. al. Using options for knowledge transfer in reinforcement learning. University of Massuchusetts, Amherst, MA, USA, Tech. Rep, 1999.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv: 1610.01283, 2016.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta- reinforeement learning via probabilistic context variables. International conference on machine learning. pp. 5331-5340, 2019.
Balaraman Ravindran and Andrew G Barto. An algebraic approach to abstraction in reinforcement learning. PhD thesis, University of Massachusetts at Amherst, 2004.
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. In Twenty-Thiee Third International Joint Conference on Artificial Intelligence, 2013.
Stephane Ross and Andrew Bagnell. Agnostic system identification for model-based reinforcement learning. arXiv preprint arXiv: /203. 1007, 2012.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image. arXiv preprint arXiv:1611.04200 2016.
Sosale Shankara Sastry and Alberto Isidori. Adaptive control of linearizable systems. IEEE Transactions on Automatic Control, 34(1 121333111331111, 1989.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint /07.:4777 2017.
12

---

Oliver G Selfridge, Richard S Sutton, and Andrew G Barto. Training and tracking in robotics. In IJCAI, pp- 670 672, 1985.
Alexander A Sherstov and Peter Stone. Improving action selection in mdp's via knowledge transfer. In AAAI, volume 5, pp. 1024-1029, 2005.
Casper Kaae Sønderby. Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferene Huszár. Amortised map inference for image super-resolution. arXiv preprint 1r10v:1610.044400 04490, 2016.
Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg. Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: On-policy maximum posteriori policy optimization for discrete and continuous control. arXiv preprint 1rXiv:1909.122 2019.
Funlade T Sunmola and Jeremy L. Wyatt. Model transfer for markov decision tasks via parameter matching. In Proceedings of the 25th Workshop of the UK Planning and Scheduling Special Interest Group (PlanSIG 2006), pp. 246-252, 2006.
Aviv Tamar. Huan Xu, and Shie Mannor. Scaling robust mdps by reinforcement learning. arXiv preprint arXiv: 1306.618,, 2013.
Jie Tan, Zhaoming Xie, Byron Boots, and C. Karen Liu. Simulation-basee design of dynamic controllers for humanoid balancing. In 2016 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 2729 2736. IEEE, 2016.
Fumihide Tanaka and Masayuki Yamamura. Multitask reinforcement learning on the distribution of mdps. In Proceedings 2003 IEEE International Symposium on Computational Intelligence in Robotics and Automa- tion. Computational Intelligence Robotics and Automation for the New Millennium (Cat. No. 03EX694), volume 3, pp. 1108-1113, IEEE, 2003.
Marko Tanaskovic, Lorenzo Fagiano, Roy Smith, Paul Goulart, and Manfred Morari. Adaptive model predictive control for constrained linear systems. In 2013 European Control Conference (ECC), pp. 382-387. IEEE, 2013.
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Louis C. Tiao, Edwin V Bonilla, and Fabio Ramos. Cycle-consistent adversarial learning approximate bayesian inference. arXiv preprint arxiv: 1806.01777, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Picter Abbeel. Domain ran- domization for transferring deep neural networks from simulation the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23 IEEE, 2017
Emanuel Todorov. Linearly-solvable markov decision problems. In Advances in neural information processing systems, pp. 1369-1376, 2007.
Mare Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th annual international conference machine learning, pp. 1049-1056, 2009.
Masatoshi Uehara, Issei Sato, Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Generative adversarial nets from a density ratio estimation perspective. arXiv preprint arXiv: 2016.
Anirudh Vemula, Yash Oza, Andrew Bagnell, and Maxim Likhachev. Planning and exccution using inaccurate models with provable guarantees. arXiv preprint arXiv:2003.00 043994 2020.
Paul Werbos. Neural networks control and system identification. In Proceedings of the 28th IEEE Conference on Decision and Control., pp. 260-265. IEEE, 1989.
Martha White. Unifying task specification reinforcemeet learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp- 3742-3750. JMLR. org, 2017.
Grady Williams, Andrew Aldrich, and Evangelos Thcodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint arXiv:1509.01 009900144 149, 2015.
Björn Wittemmark. Adaptive dual control methods: An overview. In Adaptive Systems in Control and Signal Processing 1995, pp. 67-72. Elsevier, 1995.
Markus Wulfmeier, Alex Bewley, and Ingmar Posner. Addressing appearance change in outdoor robotics with adversarial domain adaptation. In 2017 IEFF/RSI International Conference on Intelligent Robots and Systems (IROS), pp. 1551-1558. IEFF, 2017a
13

---

Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. arXiv preprint arXiv: 2017b.
Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification, arXiv preprint arXiv:1702.02455, 2017.
Bianca Zadrozny. Learning and evaluating classifiers under sample selection bias. In Proceedings of the Twenty-firs. first international conference on Machine learning, pp. 114, 2004.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-con adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 22222,22,, a..
Shaojun Zhu, Andrew Kimmel, Kostas Bekris, and Abdeslam Boularias. Fast model identification via physics engines for data-efficient policy search. arXiv preprint
2017b.
Brian D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University, 2010.
14

---

A ADDITIONAL INTERPRETATIONS OF THE REWARD CORRECTION
This section presents four additional interpretations of the reward correction, Ar.
A.1 CODING THEORY
The reward correction Ar can also be understood from the perspective of coding theory. Suppose that we use data-efficient replay buffer that exploits that fact that the next state St+11 is highly redundant with the current state and action, 8t, dz. If we assume that the replay buffer compression has been optimized to store transitions from the target environment, (negative) Ar the number of additional bits (per transition) needed for our source replay buffer, compared with our target replay buffer. Thus, an agent which maximizes År will seek those transitions that can be encoded most efficiently, minimizing the size of the source replay buffer.
A.2 MUTUAL INFORMATION
We gain intuition in the modified reward by writing the expected value of Ar from Eq. in terms of mutual information:
E Ar(e, Oe, 81+1)] (88+11 target 8t, ax) - 1 source 8t; 0t)-
The mutual information I 1$1+1; target St, az) reflects how much better you can predict the next state if you know that you are interacting with the target domain, instead of the source domain. Our approach does exactly this, rewarding the agent for taking transitions that provide information about the target domain while penalizing transitions that hint to the agent that is interacting with source domain rather than the target domain: we don't ton't want our are agent to find bugs in the Matrix.
A.3 LOWER BOUND ON THE RISK-SENSITIVE REWARD OBJECTIVE.
While we derived DARC by minimizing a reverse KL divergence (Eq. 2). we can also show that DARC maximizes lower bound on risk-sensitive reward objective (Mihatsch & Neuneier 2002)
logE, "apprpp (anpp(al)aa))) (in)) anar(ale)
exp
Σ r at)
exp Σ r(se, as) ]
8c, at)
log E. "ipaane Bxmpuusco (x)a), amx(ala)
Promnc ( (8141
8e, ar)
log By / (o' maa), amx(al>)
exp
Σ r(s), log Punge (t) log Prource (8t-1 86, 0c)
Ar(e2,ae10241)
(4)
2 E. Preaane (2)) (ola)
Σ r(s),,a) + Ar(s,, s4.11
(5)
The inequality on the last line an application of Jensen's incquality. One interesting question is when it would be preferable to maximize Eq. 4 rather than Eq. [5) While Eq. provides loser bound on the risk sensitive objective, empirically may avoid the risk-seeking behavior that can be induced by risk-sensitive objectives. We leave the investigation of this trade-off as future work.
A.4 A CONSTRAINT DYNAMICS DISCREPANCY
Our method regularizes the policy to visit states where the transition dynamics are similar between the source domain and target domain:
max E a (s,,)
a>>((ass)
Σ r(ee, ar) + log (8141 St, az) log Psouree (sc+1 |
8t, at) + (as |
DieL. (Pranc Purm))
15

---

This objective can equivalently be expressed applying MaxEnt RL to only those policies which avoid exploiting the dynamics discrepancy. More precisely, the KKT conditions guarantee that there exists positive constant > such that our objective is equivalent to the following constrained objective:
Σ r((s,,al (z) + He (or
max T TEUIAACC
amat(als)
sel
s'mp(s' (s,a)
where IARC denotes the set of policies that do not exploit the dynamics discrepancy:
Σ DKI. (proorco 82, az) Prurge((8++4 Sty at)) S 50. (6)
ARRCCC
amat(al)) s)
s d'rup(s'lann
One potential benefit of considering our method as the unconstrained objective is that it provides a principled method for increasing or decreasing the weight on the Ar term, depending on how much the policy is currently exploiting the dynamics discrepancy. We leave this investigation as future work.
B PROOFS OF THEORETICAL GUARANTEES
In this section we present our analysis showing that maximizing the modified reward + Ar in the source domain yields a near-optimal policy for the target domain, subject to Assumption To start, we show that maximizing the modified reward in the source domain is equivalent to maximizing the unmodified reward, subject to the constraint that the policy not exploit the dynamics:
Lemma B.1. Let a reward function r(s, a), source dynamics Prource (s' 8, a). and target dynamics Prarget (8 s, a) be given. Then there exists > O such the optimization problem in Eq. 2 is equivalent to
Σ r(ss 0z) + He [a, | scl
max Ep E
Eal explet
where Tho expioit denotes the set of policies that do not exploit the dynamics:
Σ DELL (Prource (8441 | St, 04) |l ( 88, s())))) acl) so..
E anv((als) Tho exploir
o' s'mp(e' (s,a)
The proof is a straightforward application of the KKT conditions. This lemma says that maximizing the modified reward can be equivalently viewed as restricting the set of policies to those that do not exploit the dynamics. Next, we will show that policies that do not exploit the dynamics have an reward that is similar in the source and target domains:
expected
Lemma B.2. Let policy π E Ino exploit be given, and let Rmux be the maximum (entropy-regularizee) return of any trajectory. Then the following inequality holds:
Σ rrss, a.) + Hx |ss]
Σ Er(s,,or) + Mnla, sil]
< 2 v c/2.
This Lemma proves that all policies in Imo exploit satisfy the same condition as the optimal policy (Assumption I.
Proof. To simplify notation, define F(s, a) = r(s, a) - log ((a s). We then apply Holder's inequality and Pinsker's inequality to obtain the desired result:
Ep
Σ F(se, ac)
Eppaae Σ F(se,
Σ (Psource(T) (rango((T))
Σ F(st, az)
F(se, as) IIProunce(T) Prarget (T)
maxx Σ r(seraa)) 2/ 2
DRL. (prource (T) || Prarger(T)
< 2mmax ve/2.
16

---

We restate main result:
Theorem 4.1 (Repeated from main text). Let TARC be the policy that maximizes the modified reggaiiy-) reward in the source domain, let π be the policy that maximizes the (unmod- ified, entropy-regutee regularredgl reward in the target domain, and assume that w² satisfies Assumption Then DARC receives near-optimal centropy-regulaare
reward on the target domain:
E Progers 7 DARC:
Σ r(ss,, az) + H(ae sa]
2 Porgers Σ E((8, 01) + H(a se]] 4.max v c/2.
Proof. Assumption guarantees that the optimal policy for the target domain, x², lies within Ho exploit- Among all policies in Il exploit. "DACC is (by definition) the one that receives highest reward on the source dynamics, so
E Possres DARC
Er(ss,au az) + H(a sel 2 Ep Parme
Σ r(s,, a.) + Hlae s
Since the both DARC and n² lie inside the constraint set, Lemma B.2 dictates that their rewards on the target domain differ by at most 2R /c22 2 from their source domain rewards. the worst case, the reward for DARC decreases by this amount and the reward for x' increases by this amount:
Repeacee DiARC
Σ r(01,a)) (4)++ H[a 8t]
S E Expger -TAKC
Σ r(se, 01) H(aa se]
2max v 6/2
Ep
Σ r(e, az) + H|a sel
2 Bx
Σ r(s az) + H(ac 8:] 2m v /e/2
Substituting these inequalities the LHS and RHS of Eq. B and rearranging terms, obtain the desired result.
C THE SPECIAL CASE OF AN OBSERVATION MODEL
To highlight the relationship between domain adaptation of dynamics versus observations, we now consider a special case. In this subsection, we will assume that the state st & (zt. 0₂) is combi- nation of the system latent state z, (e.g., the poses of all objects in scene) and observation or (e.g., a camera observation). We will define q(o, 24) and p(o; | =x) as the observation models for the source and target domains. In this special case, we can decompose the KL objective (Eq. 2 into three terms:
DK (p)) -gg Σ r(st, at) + Hx | st]
log Purget (0: 21) - log Psource (or 24)
MaxEnt objective
Observation Adaptation
log Pharget (Ze+1 Rt, (4)
log Promrce (Ze+1 Zt,
Dynamies Adaptation
Prior methods that perform observation adaptation (Bousmalis et al. 2018 Gamrian & Goldberg. 2018) effectively minimize the observation adaptation term but ignore the effect of dynamics. In contrast, the Ar reward correction in our method provides one method to address both dynamics and observations. These approaches could be combined; we leave this future work.
Tiao et al. (2018) show that observation adaptation using CycleGan Zhu et al. 2017a) minimizes Jensen- Shannon divergence. Assuming sufficiently expressive models, the Lensen-Shanoor divergence and the reverse KI. divergence above have the optimum.
17

---

broken reacher
broken half cheetah
broken ant
half cheetah obstacle
Reward (Target)
-25
-13
M
0.25
Classifer
0.00
24 Erw. Stops (Target)
0.5
0.
in
10
ERV, Stops (Torget)
Env. Stops iTangoti
Env. Staps (Targeti
Figure II: Importance of using two classifiers: Results of the ablation experiment from Fig. (left) on all environments.
broken reacher
broken half cheetah
broken ant
halff cheetal obstacle
Reward (Target)
2- 0.1
1.0
0.2
2.4
0.6
0.
1.0
24
06
0.
1.00
12
06
D6 0,5
10
Steps (Targel]
Env. Steps (Target)
Env, Steps (largel)
Ery, Sleps (Targel)
Figure 12: Importance of regularizing the classifiers: Results of the ablation experiment from Fig. 7 (right) on all environments.
D ADDITIONAL EXPERIMENTS
Figures 12 show the results of the abla-
tion experiment from Fig. 7 run on all environ- ments. The results support our conclusion in the main text regarding the importance of using two classifiers and using input noise. Figure T3 is copy of Fig- from the main text, modified to also show the agent's reward on the target do-
Reward Correction (Ar)
10.0
without år
Training
Training with Ar
5000
Rewar (Trget))
4000
-25
-7.
main. We observe that the reward does not start increasing until we start using DARC.
0.2
0.4
0.5
1.0
Training Steps
165
Figure 13: Copy of Fig. Goverlaid with the target domain reward.
E EXPERIMENT DETAILS AND HYPERPARAMETERS
Our implementation of DARC is built top of the implementation of SAC from Guadarrama ct al. (2018). Unless otherwise specified, all hyperparameters are taken from Guadarrama et al. (2018). All neural networks (actor, critics, and classifiers) have two hidden layers with 256-units each and ReLU activations. Since we ultimately will use the difference in the predictions of the two classifiers, we use a residual parametrization for the SAS classifier (target St. 8t+1). Us- ing ( des 81+1), JSA (St, ax) R² to denote the outputs of the two classifier networks, compute the classifier predictions as follows:
q:aa St, az) = SOFTMAX(fSA(St,,)) at))
40sss St, Ct, 8(+1) SOFTMAX(fSAS(s; at, 81+1) + fsA(se,
For the SAS classifier we propagate gradients back through both networks parameters, OSaS and Osa- Both classifiers use Gaussian input noise with a Optimization of all networks done with Adam (Kingma & Ba, 2014) with a learning rate of 3e-4 and batch size of 128. Most experiments with DARC collected step the target domain every 10 steps in the source domain (i.e., r = 10). The one exception is the half cheetah obstacle domain, where we tried increasing beyond 10 to 30, 100, 300, and 1000. We found a large benefit from increasing r to 30 and 100, but did not run the other experiments long enough to draw any conclusions. Fig. uses 30 for half cheetah obstacle. We did not tune this parameter, and expect that tuning it would result in significant improvements in sample efficiency.
18

---

We found that DARC was slightly more stable if warm-started the method by applying RL on the source task without Ar for the first twammup iterations. We used twarmup = le5 for all tasks except the broken reacher, where used warmup 2e5. This discrepancy was caused by typo in an experiment, and subsequent experiments found that DARC is relatively robust to different values of wwarmup: did not tune this parameter.
E.1 BASELINES
The RL on Source and RL on Target baselines are implemented identically to our method, with the exception that Ar is not added to the reward function. The RL on Target (10x) identi- cal to RL on Target, with the exception that we take 10 gradient steps per environment inter- action (instead of 1). The Importance Weighting baseline estimates the importance weights as Pharget 8t++1 8t, /e) (8t+1 St, 0t) s exp(Ar). The importance weight is used to weight transitions in the SAC actor and critic losses.
PETS (Chua et al., 2018 The PETS baseline is implemented using the default configurations used by (Chua et al., 2018 for the environments evaluated. The broken-half whw.hallffhhetttheee en- vironment uses the hyperparameters used by the hal falchetta environment in (Chua et al.) 2018). The broken-ard environment uses the same set of hyperparameters, namely: task hori- zon 1000, number of training iterations 300, number of planning (real) steps per iteration 30, number of particles to be used particle propagation methods = 20. The PETS codebase can be found at httph::///// http//gittttt com/
MBPO Janner al., 2019 We used the authors implementation with the default hyper- parameters: https: github..u.... com/ JannerM/ h//pppo We kept the environment config- urations same as their default unmodified MuloCo environments, except for the domain and task name. We added our custom environment xmls in mipp/enn/asset.. and their corresponding environment python files in the mbpo/ hevp//en////// folder. Their static files were added under vironments in the init file under mbpo_odrl wtwpoddrr////////o//oooeoooooeet htt///bppp//////ee hettleeeninng/eviirnnneee max max_episoceieooo www.episodeeeco www.episodeesstess=10000
folder,
These environments can be registered gym or can be initialized directly in aaaapetssssss /gym adapter. py. We set the time limit to for the Broken Half Cheetah, Broken Ant and Half Cheetah Obsta- cle environments and to 100 for the Broken Reacher environment.
F.2 ENVIRONMENTS
Broken Reacher This environment uses the 7DOF robot arm from the Pusher environment in OpenAl Gym. The observation space the position and velocities of all joints and the goal. The reward function
1
1
r(s, a)
2 send etfector Sgoul
10 |la
and episodes 100 steps long. In the target domain the 2nd joint (O-indexed) is broken: zero torque is applied to this joint, regardless of the commanded torque.
Broken Half Cheetah This environment is based the HalfCheetah environment in OpenAI Gym. We modified the observation to include the agent's ' global coordinate so the agent can infer its relative position the obstacle. Episodes 1000 steps long. In the target domain the Oth joint (0-indexed) is broken: zero torque is applied to this joint, regardless of the commanded torque.
Broken Ant This environment is based on the Ant environment in OpenAl Gym. We use the standard termination condition and cap the maximum episode length at 1000 steps. In the target domain the 3rd joint (0-indexed) is broken: zero torque applied this joint, regardless of the commanded torque.
In all the broken joint environments, choose which joint break computing which joint caused the "RL on Source" baseline to perform worst on the target domain, as compared with the "RL on 'Trgget" baseline.
19

---

Half Cheetah Obstacle This environment is based the HalfCheetah environment in OpenAI Gym. Episodes are 1000 steps long. We modified the standard reward function to use the absolute value in place of the velocity, resulting the following reward function:
r(s,a) = vel At ||all2.
where 8x vel is the velocity of the agent along the forward-aft axis and A = 0.01 is the time step of the simulator. In the target domain, we added wall -3m, roughly meters behind the agent.
Humanoid Used for the experiment in Fig. we used a modified version of Humanoid from OpenAl Gym. The source domain modified this environment to ignore the default termination condition and instead terminate after exactly 300 time steps. The target domain uses the unmodified environment, which terminates when the agent falls.
E.3 FIGURES
Unless otherwise noted, all experiments were run with three random seeds. Figures showing learning curves (Figures
and 12) plot the mean over the three random seeds, and also plot the results for each individual random seed with semi-transpaan lines.
E.4 ARCHERY EXPERIMENT
We used a simple physics model for the archery experiment. The target was located 70m North of the agent, and wind was applied along the East-Wes West axis. The system dynamics:
f N(p 1. = in the source domain 1~~N==0,0 = 0.3) in the target domain
8t-1 = 70)) + // (0)
We trained the classifier by sampling the source domain and 10k episodes in the target domain. The classifier was neural network with 1 hidden layer with 32 hidden units and ReLU activation. We optimized the classifier using the Adam optimizer with learning rate of 3e-3 and batch size of 1024. We trained until the validation loss increased for 3 consecutive epochs, which took 16 epochs in our experiment. We generated Fig. 4 by sampling 10k episodes for each value of and aggregating the rewards using (()) log Ep(s'10) (exp(r(s'))] We found that aggregating rewards by taking the mean did not yield meaningful results, perhaps because the mean corresponds to a (possibly loose) lower bound on (see Appendix A.33)
2[- (measured in degrees) for 10k episodes in
20


Reviews:
[{"review_id": "nRW6psxKExh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThis paper proposes a method for domain adaptation in RL where the source and target domains differ only in the transition distriubtions. A theoretical derivation based on RL as probabilistic inference is presented that starts with the objective of matching the desired distribution of trajectories in the target domain with the distribution achieved by the policy in the source domain. The final objective appears as a modification to the reward function while training in the source domain and is implemented easily with just two binary classifiers that predict the domain given either state-action or state-action-next-state tuples. Theorem 4.1 provides a theoretical guarantee on the performance of a policy trained on such a modified reward in the source domain by giving a bound on the performance in the target domain, under a very mild assumption that the optimal policy on the target domain achieves similar rewards when put in the source domain. Experiments are presented that show improved performance in terms of rewards vs experience on target domain on environments such as broken reacher, broken ant, etc (where the target domain has some \"broken\" component). Further, it is also shown that the reward modification on source visually matches the reward expected in target (Fig 4), that without the reward modification the policy usually exploits the source domain's transitions which cannot be exploited in the target domain, and finally, that safety emerges from the proposed objective.\n\n## Strengths\n- The theoretical bound presented in Theorem 4.1 is strong given that Assumption 1 is quite mild. In fact, Assumption 1 is trivially met in most of the environments used in Fig 6, e.g.: for half cheetah obstacle, the target policy does not run in to the obstacle and hence will get the same reward in the source and target environment.\n- The final form of the objective as a modification to the reward is a simple and easy to implement objective with the two binary classifiers. Further, the max-entropy derivation makes it applicable to any maximum-entropy algorithm.\n- The empirical evidence shows that the proposed method has very similar performance to the RL on target baselines while improving over other domain adaptation baselines in four continuous control environments.\n\n## Weaknesses\n- The variety in the source-target domain shift seems limited in the experiments shown in Figure 6. Three tasks have the crippled-limb setting and one task has an obstacle in the target domains. The introduction started off by giving some great examples such as aggressive driving failing to work on target domains with icy roads. It would have been great to see application of this method on more subtle but dangerous changes to the target domain, similar to the car driving example. For example, reducing friction on the HalfCheetah or Ant environments -- it would be quite interesting to see what strategies develop in the source domain; does the agent take shorter steps to staty in contact with the floor?\n- The assumption that the rewards for the target and source domains match is suitable for the choice of environments in Section 6 but seems quite limiting to a more general setting where certain rewards may never be observed in the source domain. Further, there is an implicit assumption that the source domain is \"free\" and the target domain is \"restrictive\" (e.g. broken ant, obstacle halfcheetah). It would be good to discuss the rare but still possible case where the opposite is true, the target domain allows for more \"free\" movement.\n\n## Other issues/comments\n- Theorem 4.1 has a crucial typo: $p_{\\text{source}}$ is mentioned on the right hand side but it should actually be $p_{\\text{target}}$. The appendix (Theorem B.3) has the correct version of this theorem statement.\n- In Theorem 4.1, the reward-maximizing (entropy-regularized) policy in the target domain is said to satisfy Assumption 1. To confim, does this mean that, in the target domain, the policy that maximizes the entropy-regularized reward objective is the same as the policy that maximizes rewards without the maximum-entropy constraint?\n- Minor typo in Section 5, Algorithm Summary: $q_{\\theta_{SAS}}$ should be $q_{\\theta_{SA}}$ in the second line.\n- I don't understand the transparent colored lines in Figure 6, are they supposed to represent error bars or are they the non-smooth version of the darker colored lines?\n- Typo/stray sentence in the last para, last line of Section 1.\n\n## Feedback to authors\n- To further study the usefulness of this approach (as future work perhaps), it would be good to apply it to an offline RL setting where a finite set of \"good\" trajectories from the target domain are provided and no further trajectories can be collected during training. Only during test time is the agent then deployed in the target domain to measure performance (and possibly safety).\n\n\n--------------------------\n## Post-rebuttal Update\n\nThe authors have shown new experiments on icy environments that show good results for the proposed method (DARC). This directly addresses my point (and recommendation) about trying out experiments similar to the aggressive driving on icy road example that was mentioned in the introduction. Having read through the other reviews and responses by the authors, I feel that most major concerns have been addressed. As such I am inclined to increase my score from 7 -> 8, recommending acceptance of the paper and entrusting the authors to include the new experiments in the main paper.\n\nA minor note: My second point under \"Other issues/comments\" section was not answered in the rebuttal. I hope the authors can clarify this in the future either in the main paper or appendix.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eGPsaj3lVj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Summary\n\nThe paper introduces DARC, a domain transfer algorithm motivated by maximum entropy RL. By introducing classifiers for the target and source domain, the reward function in the source domain can be modified such that it restricts the behavior of the optimized policy to transitions that reflect the target domain. In this way, the method achieves good domain transfer without having an explicit model.\n\nAdditionally, the authors provide a lower bound on the performance of such a policy in the target domain. \n\n## Overall Recommendation\n\nAlthough I believe the method to be quite restricted by the assumptions that it makes, overall, I recommend acceptance, but it could be stronger after clearing up a few things.\n\n## Writeup\n\nI really like the writeup, the connection to the related literature is clear and it is clear what is the goal of the method. The method is explained well and I didn't have any problems following the text.\n\n## Pros\n\nThe method is very intuitive and theoretically well justified. The authors provide experiments and compare against different model-free settings and a model-based baseline. I also like the connection to safety, namely that this method ensures safety by enforcing behavior that is far away from violating the safety constraints.\n\n## Cons\n\nThe experiments are still a bit dissatisfying, because I would like to see an experiment where transfer happens to the real system. Also, I can see a benefit in improving the simulators or the source domain to reflect the target domain, I don't think that taking the source domain as-is is the way to go, because this is also a waste of data from the target domain. \n\nI take note that data collection from the target domain is nevertheless necessary, in order to fit the classifiers. Might as well improve the simulator(model) in the source domain.\n\nAssumption 1 is quite restrictive.\n\n## Comments/Questions\n\nThe assumption on the reward maximizing policy from the target domain is quite strong, this also translates to an assumption about how much the dynamics are allowed to change. It would seem more satisfying to me to take into account the transition dynamics errors in deriving the bound, such as done in MBPO[1]. \n\nI believe that the way you chose to construct your experiments reflect your assumptions nicely. For the broken cheetah/ant/reacher, a policy that works with less degrees of freedom would more-or-less obviously work with more (satisfying assumption  1). I wonder how would your method perform with not so obvious changes in dynamics - changes in mass, friction etc, perhaps even adversarial changes in dynamics? \n\nNormally, when doing sim2real, the dynamics change in a different way than what is shown in the experiments. \n\nThrowing away data from the target domain and not improving the source domain seems like the wrong thing to do,  ideally you would reuse the data from the target domain to improve the source domain.\n\nFurthermore, as far as I know,  MBPO  doesn't work on transfer, but actually learns the simulator for the environment (the model). Obviously if the data for fitting the model only comes from the source domain, the model would overfit to the source domain but  stick to the regions that it's certain about. The data from the target domain should be reused  for MBPO, or it should be given the same chance as RL trained on target.  Was this done in this way? The results for MBPO seem a bit too disappointing to be true.\n\nA further separating thing is that, you assume that you basically have a simulator that \"encompasses\" the target domain, i.e. policies good in the target domain are still valid in the source domain, this is not needed by MBPO or the model-free variants, and it obviously helps the method a lot. In the broken environments case, if you would switch the target and source domains and then evaluate against the baselines, I believe that the baselines would be better or at least on-par.\n\n-------\n\n[1] Janner, Michael, et al. \"When to trust your model: Model-based policy optimization.\" Advances in Neural Information Processing Systems. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MVTJ_g2ZaOH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper introduces DARC, an RL approach that aims to transfer from a source environment to a target environment with different dynamics. DARC utilizes an adversarial-like reward that distinguishes between source and target domains. This reward is used to augment the environmental reward. DARC is evaluated against several baselines in mujoco tasks. \n\n\nOverall, the paper is well-written and the motivation is clear. The approach seems reproducible, as pseudocode is provided along with training details and hyperparameters. The technique seems simple and easy to implement. The paper also includes some theoretical guarantees for the approach. \n\n\nHowever, I believe this paper is borderline and I lean towards reject for a few reasons. The quick summary is that stronger baselines could have been included, it is unclear how well the approach can handle large (or even medium) differences in dynamics, and the approach does not seem robust to choices of source and target domains.\n\n\nWhile the method was evaluated against several baselines, I believe there are stronger methods it could have been compared to. In particular, there are quite a few imitation learning approaches that aim to learn in environments with different dynamics than the demonstrator, which is similar to the idea of transfer [1-3]. Each of these approaches has been shown to work in mujoco environments with disabled joints and hence would be good baselines. While MATL is a reasonable comparison, it was not evaluated in such environments in the original paper.\n\n\nFurthermore, using adversarial-based rewards to classify whether a trajectory is from the agent or source is not a new concept. It originated from GAIL [4] and has since seen several variants, including works that learn across different variants of the domain [5-7]. The paper should be more clear in expressing that domain classification in reinforcement learning is not a new concept.\n\n\nThe requirement that $P_{target}(s_{t+1}|s_t, a_t) > 0$ ⇒ $P_{source}(s_{t+1}|s_t, a_t) > 0$ is a weak point of the approach. It is unclear if the lower limit is 0 or some larger number in practice. This is important given the claim that DARC is for transfer in environments with different dynamics.  If the dynamics are very different, then I would expect the discriminative reward to do poorly because it would be easy to distinguish between source and target transitions. An interesting experiment would be to see how the discriminator behaves as the discrepancies in dynamics increases. \n\n\nFinally, it seems that one needs to get a bit lucky in the source and target choice. The example shown in fig 3. shows that if there were an obstacle in the target domain and no obstacle in the source domain, then the agent would need to go around this obstacle. But if the obstacle were in the source domain, I would expect the approach to be suboptimal in the target domain since the reward function would encourage it to go around an invisible object. \n\n\nSome concrete steps for improving the paper would be to include more discussion as well as comparisons to approaches that seem equipped to handling different dynamics, demonstrating the impact of differences in dynamics on the approach, and making the limitations of the approach more clear.\n \n\nOther comments/thoughts:\n\n\nThe acronym DARC is introduced in the theorem. It may be better to introduce beforehand\n\n\nWhat does the environment reward in Fig. 7 look like? It would be interesting to see a similar jump in return\n\n\nThe introduction mentions a 111-dimensional ant task. Was this actually used?\n\n\n“Our method performs 10× more gradient updates per environment step in the source domain” -- I don’t think this is explained in the paper?\n\n\nWhy does DARC sometimes have higher asymptotic reward than training from scratch?\n \nTypos\n\n\nhigher-dimensional tasks. broadly-applicable approach for learning from inaccurate models.\n\n\nthat most prior work -> than most prior work\n\n\nthe perform quite poorly -> they perform quite poorly\n\n\nso the agent is penalized -> the agent is penalized \n\n\nThis result that\n\n\nsafegaurds -> safeguards\n\n \n[1] Adversarial Imitation via Variational Inverse Reinforcement Learning. Qureshi et al.\n\n\n[2] Learning Robust Rewards with Adversarial Inverse Reinforcement Learning. Fu et al. \n\n\n[3] State Alignment-Based Reinforcement Learning. Liu et al. \n\n\n[4] Generative Adversarial Imitation Learning. Ho et al. \n\n\n[5] Third-Person Imitation Learning. Stadie et al.\n\n\n[6] Generative Adversarial Imitation from Observation. Torabi et al.\n\n\n[7] Cross-domain Imitation Learning. Kim et al.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7xVU1n33Unx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to do domain adaptation. The idea is to modify the reward function in the source domain so the learned policy can be optimal in the target domain. This is accomplished by learning classifiers that distinguish transition in the source domain from transition in the target domain.  \n\nThe idea is introduced intuitively and justified via formal analysis and it seems to effectively solve the tasks presented, but I have a few questions that need clarification before I make a final decision.\n\n(1) The assumption that the target transition will have non-zero probability in the source transition seems to be violated in some of the examples. For example, if the action for the particular broken joint is non-zero, the transition in the target domain will be entirely different from the source domain since the simulator is deterministic. I don't see how this will satisfy the assumption.  \n\n(2)Further comment on this assumption. For the motivating example of driving on roads with different frictions. Again transition in the target domain (icy road) has almost zero probability in the source domain.\n\n(3) About Assumption 1 in the paper. What is $\\epsilon$? What is $R_{max}$? I see this is defined in the appendix, but please define it in the main text.  And this assumption seems to become lemmaB2 (or look very similar to it), this is quite confusing. And for LemmaB2, there seems to be a typo in the first inequality, I assume the second p_source should be p_target, otherwise I can't see the difference between the two expectation. And what is the $p$ and $q$ in the first line of the proof? \n\n(4) Further question on the assumption, since R_max is the maximum total reward possible for a trajectory, this can be a large number (for example, for half cheetah it is something like 8000). I don't see why this assumption is meaningful.\n\n(5) For the half cheetah obstacle examples, the problem is not a MDP anymore since the obstacles is placed at some specific position and the policy doesn't have access to the position information as far as I know.\n\n(6) The experiment for humanoid standing, while interesting, seems unrealistic. In real world scenario, the robot should at least be rewarded an alive bonus for avoiding the termination condition.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to domain adaptation in reinforcement learning. The main idea behind this approach, DARC, is to modify the reward function in the source domain so that the learned policy is optimal in the target domain. This is achieved by learning a classifier that learns to discriminate between the data from the source domain and those from the target domain. 

Overall, reviewers appreciated the intuitiveness of the approach as well as its formal analysis. They had some concerns with respect to experiments, which was sorted out in the author response period. Given the overall positive reviews, I recommend accepting the paper. 

Decision: Accept (Poster)