Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation
Authors: Yihong Guo
Yixuan Wang
Yuanyuan Shi
Pan Xu
Anqi Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: off-dynamics reinforcement learning
domain adaptation
imitation learning
dynamics mismatch
Primary area: reinforcement_learning

Abstract: Training a policy in a source domain for deployment in the target domain under a dynamics shift can be challenging, often resulting in performance degradation. Previous work tackles this challenge by training on the source domain with modified rewards derived by matching distributions between the source and the target optimal trajectories. However, pure modified rewards only ensure the behavior of the learned policy in the source domain resembles trajectories produced by the target optimal policies, which does not guarantee optimal performance when the learned policy is actually deployed to the target domain. In this work, we propose to utilize imitation learning to transfer the policy learned from the reward modification to the target domain so that the new policy can generate the same trajectories in the target domain. Our approach, Domain Adaptation and Reward Augmented Imitation Learning (DARAIL), utilizes the reward modification for domain adaptation and follows the general framework of generative adversarial imitation learning from observation (GAIfO) by applying a reward augmented estimator for the policy optimization step. Theoretically, we present an error bound for our method under a mild assumption regarding the dynamics shift to justify the motivation of our method. Empirically, our method outperforms the pure modified reward method without imitation learning and also outperforms other baselines in benchmark off-dynamics environments.

Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation
Yihong Guo', Yixuan Wang', Yuanyuan Shi', Pan Xu², Anqi Liu!
Johns Hopkins University
University of California San Diego Duke University
ywang830, aliu. (alluucss@@j])) @suuuuuuu... edu, yyshiQucsd.edd pan. wmm.xuddukeeee
Abstract
Training policy in source domain for deployment in the target domain under a dynamics shift can be challenging, often resulting performance degradation. Previous work tackles this challenge by training the source domain with modified rewards derived by matching distributions between the source and the target optimal traecctoriss. However, pure modified rewards only ensure the behavior of the learned policy in the source domain resembles trajectories produced by the target optimal policies, which does guarantee optimal performance when the learned policy is actually deployed to the target domain. In this work, we propose to utilize imitation learning transfer the policy learned from the reward modification to the target domain so that the new policy can generate the same trajectories in the target domain. Our approach, Domain Adaptation and Reward Augmented Imitation Learning (DARAIL), utilizes the reward modification for domain adaptation and follows the general framework of generative adversarial imitation learning from observation (GAIfO) by applying reward augmented estimator for the policy optimization step. Theoretically, we present an error bound for our method under a mild assumption regarding the dynamics shift to justify the motivation of our method. Empirically, our method outperforms the pure modified reward method without imitation learning and also outperforms other baselines in benchmark off off- dy-amiiss environments.
Introduction
The objective of reinforcement learning (RL) is learn optimal policy that maximizes rewards through interaction and observation of environmental feedback. However, in domains such as medical treatment [I] and autonomous driving 22. cannot interact with the environment freely the emors are too costly or the amount of access to the environment limited. Instead, we might have access to simpler similar source domain. This requires domain adaptation in reinforcement learning. In this paper, we study a specific problem domain adaptation reinforcement learning (RL), where only the dynamics (transition probability) are different in domains. This is called off-dynamics RL (315) Specifically, we focus on a problem setting in which we have limited access to rollout data from the target domain, but do not have access the target domain reward, following the previous off-dynamics work (35)
Previous work off-dynamics RL, such Domain Adaptation with Rewards from Classifiers (DARC) (3] and [6]5], focuses on training the policy in the source domain with modified reward function that compensates for the dynamies differences. The reward modification derived so that the distribution of the learning policy's s experience the source domain matches that of the optimal trajectories in the target domain. As result, their experience in the source domain will
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Policy optimization
6000
Calculatc modified reward
Generntor
Training
prngee Capu
Obtain state pairs from source
Discriminator training
Source Domai
Revart Argrentated
Reward
4000
Training Reward
Target Domain
Training
Impertance Warko
Evaluation Reward
2000
Optimal Reward in Target
(% A
Iesprtance Wrights
Modlicata
Rationl from denals
Source
1 Training Steps
Diseriinntor Training
Figure 1: (a) Training reward in the source domain, i.c. Es mac Ex, r(st, ac]]. evaluation reward in the target domain, i.e. Emnarr, Pus |,, r(s;, at)] and optimal reward target domain, for DARC in Ant. Evaluating the trained DARC policy in the target domain will cause performance degradation compared with its training reward, which should be close to the optimal reward in the target given DARC's objective function. Results of HalfCheetah, Walker2d, and Reacher are in Figure 9 in Appendix. (b) Learning framework of DARAIL. DARC Training: we first train the DARC in the source domain with a modified reward that is derived from the minimization the reverse divergence between optimal policies on target and learned policies the source. Details of DARC and the modified reward are in Section 3.1 and Appendix A.I Discriminator training: the discriminator is trained to classify whether the data from the expert demonstration (DARC trajectories) and provide a local reward function for policy learning. Generator training: the policy is updated with augmented reward estimation, which integrates the reward from the source domain and information from the discriminator. We first train DARC, collect DARC trajectories from the source domain, and then train the discriminator and the generator alternatively.
produce a trajectory distribution close to the target domain's 's optimal one. However, deploying the resulting policy in the target domain usually causes performance degradation compared to its training performance in the source domain. Figure shows the experiment result of DARC under a broken source environment setting, where the broken source environment means the value of O-index in the action of the source domain is frozen to 0, and the target environment remains intact. Consequently, existing reward modification methods will only obtain a sub-optimal policy in the target domain. Details of DARC and its suboptimality in the target domain will be introduced in Section 3.1 More details about why DARC fails in more general dynamics shift cases are in Appendix C.6
In this paper, we present an off-dynamics reinforcement learning algorithm described in Figure (b). Our method, Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) consists of two components. Following previous work like DARC [3] on off-dynamics RL, we first obtain the source domain trajectories that resemble the target domain's 's optimal ones. We then transfer the policy's 's behavior from the source to the target domain through imitation learning from observation 77). which can mimic the policy's behavior from the state space.
In particular, we consider the dynamics shift in the framework of generative adversarial imitation from observation (GAlfo) [8), and propose novel and practical reward estimator called the reward augmented estimator (RaE) ) for the policy optimization step in imitation learning.
Our contributions can be summarized as follows:
We propose the Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) algo- rithm by transferring the learned policy of reward modification approaches from the source domain the target domain via mimicking state-space trajectories in the source domain. We propose reward augmented estimator (RaE) to leverage the reward from the source domain to stabilize the learning.
We recognize limitations in the existing DARC algorithm and off-dynamics reinforcement learning algorithms with similar reward modification, which is directly deploying the learned policy to the target domain results in significant performance degradation. Our proposed algorithm mitigates this issue with an imitation learning component that transfers DARC policy to the target.
We introduce an error bound for DARAIL. that relaxes the assumption made previous works that the optimal policy will receive similar reward in both domains. Specifically, with our imitation

---

learning from the observation component, we can show the convergence of DARAIL with a mild assumption on the magnitude of the dynamics shift.
We conducted experiments on four Mujoco environments, namely, HalfCheetah, Ant, Walker2d, and Reacher on modified gravity/density configurations and broken action environments. A comparative analysis between DARAIL and baseline methods is performed, demonstrating the effectiveness of our approach. Our method exhibits superior performance compared to the pure modified reward method without imitation learning and outperforms other baselines in these environments. Code is available at htttps:/github.comm httpb/.oo/ggoyy//e hooggguooioonggy/Offt
Dynamics-Reinforcee Reinforcement seaiiorremmen--LLannnngg via-Domain
Adaptation-anl ierpration-nd--aae Augmented
2 Backgrounds
Off-dynamics reinforcement learning We consider two Markov Decision Processes (MDPs): one is the source domain Mar. defined by AS,A, R. Pac: 7), and the other one is the target domain Mig- defined by (S, A. R. Pug, 7)- The difference between them is the dynamics p. also known transition probability, i.e., Pare * Prg or Pare (#t+1|8;, a+) # Prgg (#p+1 (8t, az). In our paper, we experiment with two types of dynamics shift: 1) broken environment [3], in which the O-th index value is set to be O in action, and 2) modifying the gravity/density setting of the target environment [9]. The source and the target domain share the same reward function, i.e., T'sc (St, S14 = I'ng
(5 (St, St+1 1). All other settings, including state space S, action space A, and the discounting factor 7, are the same, We will use 7 in the derivation and analysis in our paper.
We aim to learn policy C(a|s) using interaction from the source domain together with a small amount of data from the target domain (s,, 0t 8141 Jug maximize the expected discounted sum of reward R P.,, |E, (*( at) ]] in the target domain. Note that we assume we only have limited access to the target domain transition, namely (st, at, $141) in the whole process and we do not utilize the target domain reward.
Imitation learning (from Observation) Imitation Learning (IL) trains a policy to mimic an expert policy by with expert demonstration { ao). a1)..),... or {(sp. $1). ($1, ...}. Generative adversarial imitation learning (GAIL.) 7 uses an objective similar to Generative adversarial networks (GANs) that minimizes the distribution generated by the policy and the expert demonstration. It alternatively trains a discriminator Da and a policy my to solve the min-max problem:
min maxp_ (x,,)))= log (s s') + ((s, log(1 Du 8()) XH(T)), (2.1)
where is the next state and H(TT) is the entropy of the policy Ty. Note that in our problem, we mimic the state-only expert demonstrations {(s0, $1), ($1,,))..... 52)....} instead of the expert's actions. This setting also called imitation learning from observation [8). We will further discuss why we use state observation instead of action in section 3.2 D. is the classifier that discriminates whether the state pair is from the expert Tg or generated by the policy тy. Then, the policy is trained with the RL algorithm using reward estimation log (s, s') the reward. The optimization of the Eq. 2.1 involves alternatively training the policy and the discriminator.
3 Off-dynamics RL via Domain Adaptation and Reward Augmented Imitation Learning
In this section, we present our algorithm, DARAIL, under the off- off-dynamics RL problem setting. First, we introduce DARC [3] in Section [.11 which provides the distribution target optimal trajectories in the source domain to mimic. Then, in Section 3.2 we introduce the imitation learning component through which we utilize the trajectories provided by DARC and transfer the DARC policy to the target domain. We aim to learn policy that generates the same distribution of trajectories in the target domain as the DARC trajectories in the source domain.
3.1 Off-dynamics RL via Modified Reward
DARC is proposed to solve the off-dynamics RL through modified reward that compensates for the dynamies shift [3] Here, we first introduce DARC and its drawbacks. DARC seeks to match
the policy's 's experiences in the source domain and optimal trajectories in the target domain. We
3

---

define 7 = {(11, a1); ($2,02),... a2),.., (st, a...... as trajectory. We use The to represent the trajectories generated by TT in the source domain. The policy's distribution over trajectories in the source domain is defined as:
q( (T)) = (11) Pare (824 18,, az (
(3.1)
Let πT" = a Ex. E r(s,, at)l be the policy maximizing the cumulative reward in the target domain. We use - to represent the trajectories generated by x² in the target domain. Given the assumption that the optimal policy xt in the target domain is proportional to the exponential reward, i.e., I (ax)ss (8)) exp(Ee r(se, at)), the desired distribution over trajectories in the target domain is defined as:
-TT & (1() II Pog ( ar) x exp (E, ((t;,
(3.2)
DARC policy can be obtained by minimizing the reverse KL divergence of p(T and ((T)));
minz, (gllp) min E.g. E(r(st, ax) + Ar(s), at 8t+1) Hro [at]st] +
(3.3)
where (s, des 81+1) :- log Pog (5 1se, ax) log Pare Pa ($141 (ec, ar) and is partition function of p(r=7), which is independent of the dynamies and policy. The Ar(st, 0t; 8t+1) can be calculated through the following procedure: i), train two classifiers p(trg] St, 4z) and p(trg] St; Ot, 81+1) with
cross-entropy loss LcE: ii), Use Bayes' rules to obtain the log (po( (Pule:==//er_e)
Details are in Appendix C.I Eq. (3.3 shows that DARC be obtained via maximum entropy algorithm with
modified reward mmodified = r(5t 02) + Ar(st, at, $1+1) at every step.
As the dynamics shift exists, TDARC will not recover the optimal policy π², and deploying the DARC in the target domain will usually suffer from performance degradation due to the dynamics shift, as shown in Figure [[a and Figure in Appendix. However, in the source domain TDARC resembles those optimal trajectories in the target domain. Given the property of 7 we propose to use imitation learning from observation with Thanc as expert demonstrations to transfer DARC to the target domain. The new policy in the target domain should behave similarly (generate similar trajectories) as DARC in the source domain.
However, DARC matches the distribution of T and 7 "Toaac"
3.2 Imitation Learning from Observation with Reward Augmentation
In this section, we present the Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) method, which mitigates the problem of DARC via imitation learning from observation. As described in Section 3.I. TDARc resembles the target optimal trajectories, and we want to transfer DARC's s behavior to the target domain. A natural way to tackle it is utilizing imitation learning to mimic the expert demonstration rare STARC- Following 7 8, the objective can be formulated as:
min. maxp.. [Epm. Σ, log Du(se, + E (( DARC
[, log(1 - (st, 8--1))]}- (3.4)
where Du, is the discriminator in the generative adversarial imitation learning and Ç is the policy to be learned in the target domain. In the objective function Eq. (3.4). the (t+1) pairs are from the target domain, while we do not have much access to the target domain. Alternatively, we can use the (St, $141) pairs from the source domain and re-weight the transition with the importance sampling method to account for the dynamics shift. The objective with data rolled out from the source domain, and the importance sampling is as follows:
minę maxD,, { [Epse.< E, p(81, 11) log (s ((4,5 + E (((
[ log(1 (w 8141 })]}
DARC
(3.5)
where p(st, st-1) Pau (0t41) is the importance weight. Note that we do the generative adversar- 1e
ial imitation learning from only state observations (GAILfo) with (st. st+1) [HI instead of (s az). This is because we aim to learn a policy to produce the same trajectory distributions in the target as the ones Tdarc produces in the source domain, despite the dynamics shift, rather than mimicking the policy. Mimicking the (st, pairs will recover the same policy as DARC, and deploying it to the target domain will not recover the expert trajectories due the dynamics shift.
This objective Eq. 3.5) can interpreted as training the discriminator Da, to discriminate whether the (st, st-)) generated by in the target domain matches the distribution of DARC trajectories

---

in the source domain using data rolled out from the source domain with Ç and importance weight. Then, after the discriminator is fitted, the policy can be trained with the reward estimator RAE with model-free RI. The objective is:
max; Ep [Er RAE St+1]],
(3.6)
where RAE is defined as follows:
RAE 8141) log (st, 8(+1) + p(s St+1 (Tsco)) st+1) + log Du St41)).
(3.7)
Here the (st, 8t-1) is the reward obtained from the source domain, which is the same as the reward from the source domain, i.e. Firy(e (8t,,,,,1 St+1). In imitation learning. the - log (st 81+1)) can be viewed max; B 1,, log (s,, St+1)]. 1)]. So (3.5 can be viewed as learning reward function for the training of Ç. However, as the dynamics shift exists, the estimation of the log (sr, 8t+1)
a local reward function for the policy optimization step and the objective is
could be biased, which similar to the case in off-policy evaluation (OPE) [12-66] when training reward estimation on biased data. As we have access to the source domain and can obtain the reward from the rollout, we are motivated to use both the reward estimation log (s,, $c-1) and the ground truth reward in the source domain (St, S1+1) that could have better reward estimation than - log (st, $t-1) under dynamics shift. The RAE here can be viewed as using base estimator of the reward and use Tax (St, St+1) and importance weight
log (st, St+1) p(8,, St+1) to correct it. This correction idea is similar to the doubly robust estimator (DR) [2] in OPE. The DR estimator combines the reward estimation f and the importance weighted difference between true reward r and f. Specifically, the DR method takes the reward estimation & as a base estimator and applies the importance weighting to the difference between true reward and which is p(r - f) term, to correct the bias of the f, where is the importance weight.
Algorithm Domain Adaptation and Reward Augmented Imitation Learning (DARAIL)
1: Initialize: source and target environments Mar and Mig; replay buffers for source and target
transitions, (DTac Dí Dí initial parameters for the two classifiers 0 (OSA: ASAS): initial policy (TDARC, S); initial discriminator Dw, ratio of experience from source target, ratio k of update frequency of generator vs. discriminator.
2: TDARC Call DARC [3]
training expert policy
Reward Augmented Imitation Learning
3: DETAK <- DDRAR U (JOOLLOU(TTARR Mare)
4: for = 0, ...T do
5: Dàe +- Dec UROLOOUT(C, Marc)
6: if mod = 0 then
7: D + DDg U JOLLLOUT(S,
8:
end if
9:
if / mod h == 0 then
Da IL.(DruNe Dr,,, L), where L is from Eq. 3.5 11: end if
update discriminator
0 argmin LCE(Dáe D
update classifiers by cross-entropy loss >> reward augmented estimator > update generator
13: Calculate RAE from (37))
14:
< SAC(S, Dic RAE)
end for
16:
Output:
Our Algorithm The DARAIL is shown in Algorithm|] [ which consists of two steps: the first step, Line 2 in Algorithm I is the training of TDARC, and the second step is imitation learning with the reward estimator in Eq. (37)). In Lines 6-8, we roll out the target domain transition (st, ats St41) to calculate the importance weight. Here, will not collect the target domain reward. In Lines 9-11, we update the discriminator based on Eq. (3.5). In Line 12, we train the two classifiers p(trgl (rrglls, az) and p(trg) St, de, $1+1) with cross-entropy loss Leg and Bayes' rules similar to Ar Oe, St+1) in DARC as mentioned in Section The details are in Appendix C.I Lastly, we calculate the RAE in Line 13 and update the generator (Soft Actor-Critic (SAC) (17) with RAE in Line 14.
Note that in Lines 6-7, we roll out from the target domain, but the amount of is significantly smaller than the source rollouts. In experiments, we roll out from the target domain every 100 steps of
5

---

source domain rollouts, which is 1% of the source domain rollouts. Further, even though DARAIL requires more target domain rollouts than DARC as is required train DARC first and then perform the imitation learning step, the advantage of DARAIL does not solely come from the more target samples. Because, in DARC, increasing the training step or target domain rollouts will not further improve its performance due to its inherent suboptimality, which is shown in table 1T and 12 in Appendix with the same amount of target domain rollouts.
4 Theoretical Analysis of DARAIL
Let πr = argmax, En, 1 r(st, ar)] be the optimal policy maximizing the cumulative reward in the target domain and Ç be the policy learned from DARAIL. Now, we provide an error bound for DARAIL.. Details of the proof are deferred Appendix B
Theorem 4.1. Let m be the number of the expert demonstration and R(m) be the empirical Rademacher complexity. Let B be the error bound of DARC in the source domain, i.e. Epm.. [E, rSt,, at) + H[ay st]] Branc E. rst,, az)l S. B and W be the upper bound of the importance weight, i.e. p(st, S1+1) S W, (St, $141). ). Let discriminator class D be a A-bounded function, i.e. [Da(St, ((+1)) S A given any (St, 81+1). ||ll| measures the richness of the discriminator represent the ground truth reward as defined in Appendix B.2 dp is a defined neural network distance between the (st, $1+1) distribu- tions generated by the TAARC and oe defined in Appendix B.I] Given the empirical training error of
1 E. DED mi 2im1 o,D(st,
Ep Frane
the imitation learning, i.e. d TTsaac FE
inffç dD(f (f TTARK F(T)
S L.V. V.V.E E (0, 1), with probability
at least 8, we have
E P EE D(s,, E Drok (Er(s))))
E BemAADRR Ee (() + Hlaelsell Exiie,Taae EE, (((),a)))
DARCC Error Beand Source
+ +IIllD|ē (ē+ + inf (f
2R
(m)
2WR (m)
+ +6W + 1)A1 log(4/0)/2m
TWDARC
(2.1) Approsimation Error
(2.2) Estimaaion Ervor
(2) Imitation Learning Error Bound
Remark 4.2. Our error bound depends on (1) the DARC error bound in the source domain and (2) the imitation learning generalization error, where (2) is further decomposed into (2.1) approximation error and (2.2) estimation error. This bound demonstrates how the two important components in our proposed approach contribute to a good performance. Firstly, we would want a well-trained policy on the source reduce (1), which can be achieved by good policy learning algorithm and well well-trained classifiers for reward modification. Secondly, we utilize imitation learning from observation to transfer the experience to the source. (2.1) depends the upper bound of the importance weight, which can be decreased with a richer policy class or when the dynamics shift becomes smaller. Additionally, a betler imitation can be also achieved by increasing the complexity of the discriminator function class and the number of sumples, which pushes (2.2) to be smaller.
4.1 Comparison with the Analysis of DARC
As discussed in Section 3.1 the DARC algorithm 3 trains a policy TDARC on the source domain via matching the distribution of trajectories generated by TDarC in the source and the distribution of the optimal trajectory in the target domain. Consequently, the learned policy daarC will be suboptimal it is directly deployed in the target domain.
In the DARC analysis, it is assumed that the optimal policy for the target domain x lies in the no exploit set defined as follows [3] Assumption 1].
(4.1) Here, the exploit set means that the experiences for any policy in this set are similar in the source and target domains. Consequently, any two policies in this no exploit set also receive similar expected rewards in the two domains, and thus the reward received by nr" in the target domain is similar to
Im exploit A [Ea=s(als) E, DKL (Pac (St41 at)llpng (1t, a:)) S c).

---

that received by TDARC in the target domain. Further, the objective function Eq. equivalent to the following constrained optimization.
3.3 of DARC is
mal explei
Σ r(st, a.) H[[a]s+]]
(4.2)
Thus, deploying the policy TDARC will not receive huge performance degradation. However, the assumption that x² exploit is large. When this assumption is violated, x'' is not good policy in the source domain, though it is the optimal policy in the target domain. Thus, the DARC poliey which only optimizes the modified reward in the source domain will have significant performance degradation, as we have empirically We also demonstrate this performance gap in Lemma A. in Appendix A when their assumption is not satisfied.
exploit is stringent and might not always be satisfied when the dynamics shift
shown in Figure T (a) and Figure 9
In contrast, our algorithm DARAIL does not assume the performance of TARCC in the source domain to be close to the performance of x² in the target domain. Instead, we only assume that the importance weight somehow bounded, meaning that the dynamics shift is bounded. The error bound of our algorithm presented in Theorem 4.1 is controlled by imitation learning, which transfers the performance of TDARC in the source domain to that of 'T" the target domain without assuming r E IT exploit- Therefore, our algorithm can work well even in the cases shown in Figure [] and Figure where the experience of TDARC is very distinctive in the source and target domains.
Experiment
In this section, we conduct experiments off-dynamics reinforcement learning settings four OpenAl environments: HalfCheetah-v2, Ant-v2, Walker2d-v2, and Reacher-v2. We compare our method with seven baselines and demonstrate the superiority of the proposed DARAIL.
5.1 Experiments Setup
We examine our algorithm with two types of dynamies shift. 1) Broken environ- ment. Following previous work (3. we freeze the O-index value to O in action: zero torque is applied to this joint, regardless of the commanded torque. Different from DARC 33 who only test their method in intact source and broken target environment, we further test our algorithm in the broken source and intact target environment, where the source has less support than the target domain. As discussed in Section 4.1 violating the π-* Π exploit assumption leads to significant performance degradation for DARC and similar methods. When the source domain is intact, this assumption is more likely to hold and DARC can achieve near-optimal policy in the target domain. So, besides the setting in DARC, we focus on a harder problem for off-dynamics RL where DARC is prone to failure due to the violation of the assumptions in Section 4.1 Further, for the Ant and Walker2d, the source environment is broken with p = 0.8 probability, which means that with 0.8 probability, the 0-index will be set to be O, and 0.2 probability remains the original value. More details about the broken environment will be introduced in the Appendix C.3 2) Modify parameters of the environment. Besides the broken environment, we create dynamics shifts by modifying MuJoCo's configuration files for the target domain. Specifically, we modify one of the coefficients of (gravity, density) from 1.0 to one of the value {0.5, 1.5).
Dynamics Shifts:
Baselines: We first compare our method with DARC performance in the source and target domains. DARC Training and DARC Evaluation, defined Sper, 1E, r(st, a,)] and EPPLL-TDRRC >E, r(s,, a+)) respectively, represent DARC performance in the two domains. We compare DARAIL with DARC training performance as we mimic the DARC behavior in the source domain, which should receive a similar reward as the DARC training reward in the source domain. We compare with DARC Evaluation to show that our method mitigates the problem of DARC and outperforms DARC in the target domain. Further, we compare our method DARAIL with several baselines that we describe as follows. Importance Sampling for Reward (IS-R) re-weights the reward in the transition (s 11s....) with ( a2) and update the policy with reward Pra(scials..0) r(t,, a+) 18. Importance Sam-
pling for SAC Actor and Critic Loss (IS-ACL) [18 re-weights the transitions in the SAC actor and critic loss. DAIL is a reduction of DARAIL without reward augmentation. Model-based RL method
MBPO 19 uses short model rollouts branched from real data to reduce the compounding errors of inaccurate models and decouple the model horizon from the task horizon. MATL 20 uses different
modified rewards and is similar to our problem setting, except that they have access to rewards in
7

---

the target domain. Finally, we compare with generative ad adversarial reinforced action transformation (GARAT) [TO), a grounded action transformation method that uses imitation learning to modify the action that is executed in the source domain to simulate the target transitions. More details of the baselines are in Appendix C.2
Experimental Details:
We perform weight clipping to all methods that use the importance weight, including the DARAIL, DAIL, IS-R, and IS-ACL, and select the [0.01, 100] as the clipping interval for fair comparison, which works well for all methods. We also show that DARAIL is less sensitive to the importance of weight clipping in the next section. We conduct fair parameter tuning for our method and baselines, including learning rate, Gaussian noise scale, and learning frequency of the importance weight. We also tune the parameter for the imitation learning component in DARAIL and DAIL and notice that the higher update frequency tends to perform better, and experiment results are in Appendix D.2
More details are in Appendix D.4
5.2 Results
We show the results of DARAIL and DARC in Table [ and 2 for broken source and 1.5 gravity setting, respectively, And the results of other baselines are Table 3 and 4 We refer to the results on other settings in the Appendix, including the intact source and broken target environment setting and the modification of different scales of the parameters in the configuration file. We will also empirically discuss why DARC works well in the broken target setting while fails in the broken source setting in Appendix C.6.
Table 1: Comparison of DARAIL with DARC, broken source environment. DARC Evaluation DARC Training Optimal in Target
DARAIL.
HalfCheetah
4133 + 828
6995 +
8543 + 230
7067 + 176
Ant
4280 + 33
5197 + 155
6183 + 348
5357 + 79
Walker2d
2669 + 788
3896 + 523
3899 + 214
4366 + 434
Reacher
-26.3 + 3.3
-11.2 2.9
-7.2 + 1.2
-13.7 + 0.9
Table 2: Comparison of DARAIL with DARC, 1.5 gravity.
DARC Evaluation DARC Training Optimal in Target
DARAIL
HalfCheetah
653 = 142
4897 = 653
6894 491
4093 + 1021
Ant
1587 + 594
2170 + 258
5320 + 429
3472 + 771
Walker2d
257 + 28
4130 + 689
4254 + 345
4409 + 401
Reacher
-55.3 = 10.3
3.8
-8.3 + 1.3
-9.5 = 0.22
The Suboptimality of DARC and DARAIL outperforms DARC By comparing DARC Training and DARC Evaluation in Table [ and 2 we demonstrate that there is performance degradation of TDarc deployed in the target domain on all four environments. TARC reward in the target domain is about 40% lower than DARC reward in the source domain on average for broken source setting, and the degradation can be more severe in the changing gravity and density setting. Also, TDACC reward in the target domain is significantly lower than the target optimal reward. The training reward curves of DARC of the broken source environment setting are in Appendix C.5 clearly showing performance degradation when deployed in the target domain. Further, DARAIL outperforms the DARC evaluation performance.
DARAIL Outperforms Baselines We show the result of DARAIL and baselines in Table The training curves of other settings are in Appendix C.4 In all four environments, DARAIL outperforms the TDARC reward in the target domain. DARAIL also achieves better performance or the same level of rewards compared to the TDRRC in the source domain as shown in Table and 2 which is our expert policy for the imitation step. Compared with the DAIL, DARAIL has a much better performance, which demonstrates the effectiveness of the reward estimator RAg. Compared with the two important weighting methods, IS-R and IS-ACL, in broken source settings. DARAIL outperforms IS-R in four environments and IS-ACL in Ant and Walker2d. IS-ACL and DARAIL achieve similar rewards in HalfCheetah and Reacher. And in modifying configuration settings, DARAIL outperforms IS-R and IS-ACL. Our method outperforms MBPO, MATL, and GARAT in all environments.
DARAIL is Less Sensitive to Extreme Values in Importance Weights Though IS-ACL achieves comparable performance with DARAIL on some tasks shown in Table
3 it is highly sensitive to
8

---

Table 3: Comparison of DARAIL with basclines in off-dynamics RL, broken source environment.
DAIL
IS-R
IS-ACL
MBPO
MATL
GARAT
DARAIL
HalfCheetah
6402 362
6007 863
6934 231
4323 7
1538 616
5877
382
7067
176
Ant
3239 = 395
1463 + 1055
2753 + 94
2445 + 13
2006
17
3380 + 268
5357
+ 79
Walker2d
2330 + 156
3092 1 434
3881
+ 269
1012
250 1
3296 1 284
4366
434
Reacher
- 13.9 + 1.1
17.6 + 0.25
14..1 1: 0.16
-4.3
30 - 10
- -4.77 + 2.6 -13.7
+ 0.9
Table 4: Comparison of DARAIL with baselines in off-dynamics RL, 1.5 gravity.
DAIL
IS-R
IS-ACL
MBPO
MATL
GARAT
DARAIL
HalfCheetah
2666 + 2037
2718 1978
3576
312
619
337 205
3825 + 437
4093 + 1021
Ant
990 + 251
1712 + 393
2396 + 573
13
1376 + 466
1961 = 115
3472 771
Walker2d
525 + 142
1543 + 604
1369 + 705
870 $70
1419 + 489
630 +: 230
4409 401
Reacher
- -16.5 + 1.1
- 144.6 0.8
8.3
- 0.9
17.6
- 16..7 0.3
-9.5 1 0.22
we show the performance of DARAIL and IPS-ACI. on different importance weight clipping intervals in the broken source setting, and DARAIL outperforms IPS-ACL on all tasks. If the clipping interval is too large, IPS-ACL. suffers from high variance, thus harming the performance. If the clipping interval is too small, the effective information about the dynamics shift is lost. On the other hand, DARAIL is less sensitive it, which is an inherent property of our RAg. Furthermore, in Figure for IPS-ACL, the training curve for [0.001, 1000] clipping interval has a much larger variance than [0.1, 10] clipping interval, while our method does not suffer from such high variance. This also demonstrates that our proposed reward estimator RAE is a more robust estimator and less affected by the importance weight.
the clipping interval of importance weight. In Figure
2
DARC Sralkaton Thail "g S-AC.
DABC Evaluttinn
DiBC Evakation
DARC Tuining
Thairing Steps
Thiining Steps
Thiiiing Staps
Theining Staps
(a) HalfCheetah 0.11
(b) HalfCheetah
(c) Walker2d
(d) Walker2d [0.001, 1000]
[0.001, 1000]
(0.1, 10]
Figure 2: Performance of DARAIL and IPS-ACL on HalfCheetah and Walker2d under different importance weight clipping intervals. DARAIL outperforms IPS-ACL on all tasks. Table 3 IPS-ACL receives comparable performance with DARAIL with the clipping interval [0.01,100], while the performance decreases significantly with different intervals.
DARAIL's Performance on Different Magnitudes of Shifts In our broken action environments, as we create the off-dynamics shift by (probabilistically) freezing one action dimension in the source domain, we can control the off-dynamics shift magnitudes by controlling the broken probability. For the same environment, the larger the py is, the higher the probability of freezing the O-index action, thus a larger dynamics shift. We consider f = [0.2, 0.5, 0.8] for Ant, respectively and the experiment results is shown in Figure From left to right, the dynamics shift increases, we observe that the DARC performance decreases, and DARAIL outperforms DARC on all tasks.
6 Related Work
Off-dynamics RL Off-dynamics RL 3 is a specific domain adaptation 21 22 and transfer learning problem in the RL domain [23] where the goal is to learn policy from source domain to adapt to a target domain where the dynamics are different. Similar to many works in off-policy evaluation
(OPE) 12 in bandit and fffline/offfpolice RL [13 24 an importance weight approach can be used to
account for the difference between the transition dynamics with (1.
However, this method
can easily suffer from high variance due to the estimation bias of Pare (S az)
12 Another line
of method for the off-dynamics RL is through reward shaping 3 (5) DARC (3) learns a policy from
modified reward function that accounts for the dynamics shifts through trajectories distribution
matching objective. 66) proposed an unsupervised domain adaptation method with KI regularized objective, which uses the same reward modification techniques trajectories distribution matching

---

DAAL DARO TO 180
DARE EAIlaS01 DARC Traee
DAAC Taining
Thaining Steps
Training Steps
Taining
(a) Ant, pf 0.2
(b) Ant. Ps 0.5
(c) Ant, P5 0.8
Figure 3: Performance of DARC and DARAIL under different off-dynamics shifts Ant. Action 0 is frozen (set to be 0) with probability py in the source domain. From left to right, the off-dynamics shift becomes larger. As the shift becomes larger, the gap between DARC Training and DARC Evaluation is larger. Our method outperforms DARC on different dynamics shift.
objective in DARC [3. These reward-shaping methods all face the same problem: they will not recover the optimal policy in the target domain and will suffer from performance degradation in the target domain, but the policy's experience in the source domain similar the optimal policy in the target domain. Similarly, (25] proposes state-regularized policy optimization method that constrains the state distribution be similar in the source and target domain by adding a constraint term in the reward, However, this will also lead to suboptimal policy in the target domain like DARC. Different from DARC, Mutual Alignment Transfer Learning (MATL) [20] uses different modified rewards with GAN 126 to align the trajectories generated in the source and the target domain, but it requires access to the target domain reward. There is also work (27) that solves the off-dynamics RI. problem by training a distributionally robust policy in the source domain by assuming that the target domain's 's transition probability is in an ambiguity set defined around the transition probability of the source domain. Our method builds on DARC, inspired by its property in the source domain, overcoming the issues in DARC and similar methods by mimicking the DARC behavior the source domain.
Imitation Learning Imitation learning (IL) is another line of work that can be applied to off- off-dynamics problems by mimicking the expert demonstration
the target domain. Generative adversarial imitation learning. 17/28/ 30 8 31 (32), frames IL as an aecupaney-measure
measure matching or divergence minimization problem, which minimizes the divergence of the generated trajectories and the expert demonstration. Building on GAN 226] it uses the RI. algorithm as generator and classifier as discriminator to achieve this. Imitation learning from observation (/fo) 33 recently proposed to mimic the expert's 's behavior without knowing which actions the expert took. In the off-dynamics RL setting, recent work on IL under dynamics mismatch [T 36) can transfer a policy learned in the source to the target domain with minimal interaction with the target domain. However, these methods require high-quality and sufficient expert demonstrations and also the expert demonstrations might not be the optimal trajectories for the target domain, resulting in a suboptimal policy. Our method, DARAIL, transfers the DARC policy's 's behavior in the source to the target domain through imitation learning from observation that the target domain. Furthermore, we propose a novel and practical reward estimator with the signal from the discriminator and the reward from the source domain for the policy optimization.
policy will behave like the optimal policy in the
7 Conclusion
In this paper, we propose Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) for off-dynamics RI. We recognize the drawbacks of DARC and its following work with the same modified rewards function. We demonstrate that DARC or similar reward modification methods can only obtain a near-optimal policy in the target domain. We then propose to mimic the trajectory distribution generated by DARC in the source domain. Specifically, we propose a reward- augnedd estimator for the policy optimization step in imitation learning from observation. Theoretically, we established the finite sample upper bounds of rewards for the proposed method, relaxing the restrictive assumption about the optimal policy in the previous work. Empirically, we conducted experiments on four Mujoco environments, demonstrating the superiority of our method. From the safety perspective, our method avoids directly training policy in a high-risk environment. Our future work includes investigating off-dynamics reinforcement learning under safety constraints and more severe domain gaps in reinforcement learning.
10

---

Acknowledgmentt
We would like to thank the anonymous reviewers for their helpful comments. YG supported by the Center for Digital Health and Artificial Intelligence (CDHAT) of the Johns Hopkins University. PX was supported in part by the National Science Foundation (DMS-23231 12) and the Whitehead Scholars Program at the Duke University School of Medicine. AL was partially supported by the Amazon Research Award, the Discovery Award of the Johns Hopkins University, and seed grant from the JHU Institute of Assured Autonomy. The views and conclusions in this paper are those of the authors and should not be interpreted as representing any funding agency.
References
[1] Ying Liu, Brent Logan, Ning Liu, Zhiyuan Xu, Jian Tang, and Yangzhi Wang. Deep rein- forcement learning for dynamic treatment regimes on medical registry data. In 2017 IEEE international conference healthcare informatics (ICHI), pages 380-385. IEEE, 2017.
[2] B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909-4926, 2021.
[3] Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhut- dinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arXiv preprint arXiv:2006. /39/6, 2020.
[4] Junda Wu, Zhihui Xie, Tong Yu, Qizhi Li, and Shuai Li. Sim-to-real interactive recommendation via off-dynamics reinforcement learning. In 2nd Offline Reinforcement Learning Workshop Advances at NeurlPS, 2021.
[5] Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-awes aware reward augmentation in offline reinforcement learning. arXiv preprint arXiv:2203. 06662, 206662203..0666,, 2022.
[6] Jinxin Liu, Hao Shen, Donglin Wang, Yachen Kang, and Qiangxing Tian. Unsupervised domain adaptation with dynamics-aware rewards in reinforcement learning. Advances in Neural Information Processing Systems, 34:28784-28797, 2021.
[7] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[8] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprint 180 2018.
(9] Shengyi Jiang, Jingcheng Pang. and Yang Yu. Offline imitation learning with a misspecified simulator. Advances in neural information processing systems, 1088510-8520, 2020.
[10] Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems, 33:3917-3929, 2020.
[11] Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. arXiv preprint arXiv:2002. /8879, 2020.
[12] Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint 1rXiv:110.4460 2011.
[13] Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning.
In International Conference on Machine Learning, pages 652-661. PMLR, 2016.
[14] Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning, pages 9167-9176. PMLR, 2020.
[15] Tengyu Xu, Zhuoran Yang, Zhaoran Wang, and Yingbin Liang. Doubly robust off-policy actor-critic: Convergence and optimality. International Conference on Machine Learning, pages 11581-11591. PMLR, 2021.
11

---

[16] Nathan Kallus, Xiaojie Mao, Kaiwen Wang, and Zhengyuan Zhou. Doubly robust distribu- tionally robust offlpy evaluation and learning. In International Conference on Machine Learning, pages 10598-10632. PMLR, 2022.
[17] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[18] Joshua Arvind Holla. On the off-dynamics approach to reinforcement learning. McGill University (Canada), 2021.
[19] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[20] Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. In Conference on Robot Learning, pages 281-290. PMLR, 2017.
[21] Thomas Carr, Maria Chli, and George Vogiatzis. Domain adaptation for reinforcement learning on the atari. arXiv preprint arXlv:///2.00452,, 2018.
[22] Jinwei Xing, Takashi Nagata, Kexin Chen, Xinyun Zou, Emre Neftei, and Jeffrey L Krichmar. Domain adaptation in reinforeement learning via latent unified state representation. In Pro- ceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 10452-10459, 2021.
[23] Zhuangdi Zhu, Kaixiang Lin, Anil Jain, and Jiayu Zhou. Transfer learning in deep reinforce- ment learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023.
[24] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0 006643,,3,,, 2020.
[25] Zhenghai Xue, Qingpeng Cai, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. State regularized policy optimization on data with dynamics shift. Advances in neural informa- tion processing systems, 36, 2024.
[26] Ian Goodfellow, Jean Pouget-Abacee Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[27] Zhishuai Liu and Pan Xu. Distributionally robust off-dynamics reinforcement learning: Prov- able efficiency with linear function approximation. In International Conference on Artificial Intelligence and Statistics, pages 2719-27211 PMLR, 2024.
[28] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning, arXiv preprint arXiv:1710.112444 2017.
[29] Kee-Fung Kim and Hyun Soo Park. Imitation learning via kernel mean embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[30] Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. arXiv preprint arXiv: 18 1810.00821, 2018
[31] Mingxuan Jing, Xiaojian Ma, Wenbing Huang. Fuchun Sun, and Huaping Liu. Task transfer by preference-basee cost learning. In Proceedings the AAAI Conference on Artificial Intelligence, volume 33, pages 2478.2478 2019.
the
[32] Faraz Torabi, Garrett Warnell, and Peter Stone. Imitation learning from video by leveraging proprioception. arXiv preprint arXiv:1905.0933. 2019.
[33] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1118-1125 IEEE, 2018
12

---

[34] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.019: 01954, 2018.
[35] Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observation. arXiv preprint arXiv: 130.,1366,, 2019.
[36] Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning, pages 5286-5295. PMLR, 2020.
[37] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and cquilibrium in generative adversarial nets (gans). In International conference on machine learning, pages 224-232. PMLR, 2017.
[38] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems, 33:15737-15749 2020.
13

---

A Analysis of DARC
A.1 DARC Objective
4 shows the objective of DARC, which minimizes the reverse KL divergence of the trajectories generated by the TDARC in the source domain and π in the target domain. Note that the optimal policy is assumed be proportional to the exponential form of the reward, i.e. x exp(r(sr, de) de)). Given this assumption, the reverse KL divergence can be re- re-formulated to Eq. (3.3) with modified reward. So, the DARC will be optimal in the target domain but can generate trajectories in the source domain that resemble the optimal trajectories given the objective.
π
DARC
Target Domain
Source Domain
(sL, S1+1)
(St, St+1)
Minimiize Reverse KL Divergence
Figure 4: Optimization objective of DARC. DARC minimizes the reverse KI. divergence of the trajectories generated by the TDARC and optimal policy x².
A.2 DARC Error Bound
Now, we show that without the assumption n* Imo exploit in (3, the error maRc cannot be trivially bounded.
Lemma A.1. Tf $ Ino exploit the error bound of the TiARe is in the following form:
Ex Σ r(se, an) + H/a so]
Σ r(s), + H[aclse]
Epor. TTAANC
2Rmax DEL (Prrg (T), Pr (r))) Σ (IV(OROR((1s))) n" (+(s))
max Sc,0e,8411
Ar(st, St-1)
+ +2Rmaz eff2.
Proof. Lemma B.2, they show that for any policy क E Il exploirs the following inequality
holds:
B Pec Σ r(S,, ar) + Hxx
- E Eapg o
Σ r(s,, at) + Hx [alss])
< 21 v Ve/2, (A.1)
where Rmann refers to the maximum entropy-regularized return of any trajectories. However, the inequality Eq. I). only holds for TDARC, not for x*. Now, we show that without the assumption
o" E Th expliis the error could not be bounded trivially.
We start with the same decomposition. Therefore, we have
Σ r(ss, ar) + H[ayssi
Er(se r(ss ar) + M[alss
E Ex
E Pay. STaac
14

---

E Σ a() H[arss
Emecst
Σ r(se, az) ) + H[ac]sc]
f1
+ Ep Σ a(( + H[aelss]
Expas ADARC
Σ r((), Hx (anlsr)
12
+ Eu ATDARC
Er(s az) + H[arlse]
Epec TDARc
r((s) + H[aa]st]
(A.2)
In the original proof of 3 they bound the three terms based on the following idea:
For the term 11. they directly assume x² E Iho exploir and obtain 11 S 2mmaz v e/2 based on inequality Eq. (A.I). However, without the x² E 1lno exploits the upper bound is not valid. A valid upper bound should be:
11 Ep
Σ Er(se, az) + H[arlse]
- Execit
Σ r(se, ar) + H H[arss]]
(pug (T) - P (r))
Σ
r(se, ar) + +((als)))
S Rmaz Plgg (T) - Pere,or (
< 2Rnar V
(T), Prre.r- (T))-
(A.3)
If nT" E Tno exploit holds, we have D Pae, (T)) S e, which recovers the inequality Eq. (.II). it doesn't, cannot trivially bound the DKL (Pug,r (T)
For the term I2, the proof of [3] they also assume n² E Tno explots and obtain the I2 S based on the objective TDARC maximizes the reward in the source domain with TDARC 1l explott- If
T E oln exploir doesn't t hold, we can bound this term by the following inequality:
E Poe, TDARC
Σ r(s,, ar) + Ar(s,, ae S1+1) + H [a [sc]
2 E
Σ r(st, ar) + Ar(st, az s141) + H(aals1)
which is equivalent
Expc Σ (()) + H[(a(())]
E FDARC
Σ r((), H(arlsa]
E
Σ Ar(s, 8642)
Eppas
Σ ar,se,
(A.4)
Σ
max
Ar ae, S(+1).
(A.5)
SL,0
And the total variation the two policies cannot be trivially bound as well. For the term I3. we can easily bound by applying the inequality Eq. (A.I) as #DARC Tho exploir- In summary, the bound without assuming x E I exploir will be:
Expg,
Σ r(se, as) + H[aessc]
E Epmy
Σ r(s ar) + M(ayssi
15

---

S 22max V DEL (Pag (T), Psac, (T)) + Σ (TDARC (-18),, x (11=t) max Ar(st,
86618111
+ Rmaa Ve22.
This completes the proof.
B Theoretical Analysis of DARAIL
In this section, we prove our theoretical results. Definition B.1. (Neural Network Distance 38) For a class of neural networks DD the neural network distance between two state-next state distributions, The and TC is defined as
M8 sup (eeeee41)
[D(s,, ( (8) sttill}
DED
sup E (s / FBARC
s(t1)l3 [D - (p(sc- $141) s(t))))+))]
DED
FEARC
Definition B.2. (Empirical Rademacher Complexity) Given function class F, dataset X
(I1, 22, ...: Tn). i..d drawn from distribution y and random variable a defined as P(o = 1) =
P(o -1) b. the empirical Rademacher complexity is given by:
R((0) = [Eo[sup] fEF
Σ (x.)
(B.1)
Definition B.3. (Linear Span of the Discriminator) Consider span of the discriminator class: spant D) (co Σx c; co E TR D; DD, E N). Assuming the ground truth reward function lies in the span(D), then the compatible coefficient is defined as:
k
|rllp inf Σ Iel::-- :el::r=a co Σ D.D.COERDDDEDD... E.D.EER..DEDDEE
(B.2)
The compatible coefficient represents the minimum number of functions in D required to the reward function r, which means the complexity of the reward function
Lemma B.4. (GAIL Generalization), Let ADARC be the expert policy and ? be the solution of the imitation learning algorithm. Let discriminator class D be A A-bounded function, i.e. [D S14 S
A. Suppose reward function r lies in the span of the discriminator class. Given dp (f k13)) poofmmeee ( S (empirical neural network distance achieved by imitation learning). the importance weight p(s, St41) bounded by W, m the number of the expert data, then V E (0, 1), with probability at least 1 5, we have
infç dp(Fte
Σ I((
- E m Σ Er(s).a))
FoARc
log (4g(4/8) 2m
< ||rl|| inf dive Neuee: + + 27 (m) Tranc
+ 2WR (m)
+ +6W + 1)A
Proof. Given do Piuneee
T infç do dflaaamnc
18g
S we can have
d
T(
< (T Touer? T
dD(= TooRe!
+ inf
dp (fste
{ + é.
C
We prove that dp sce
urg TC
drs prre
m { has upper bound.
#
dp A
T
trg
d'
"uneee TC
try
sup DED
IP. (e
(D(8,) 8141)]
E ((o D 8t+1)]
16

---

sup (s - Fimcc (D(, s(+1)] B (oc (st s[+1)]]
DED
sup DED
E (s Toaee [D(5 S1+1)] 1)s
[s, 8c+ sc+1]]
+ sup DED
E
(1 (80,, 86+1)] - -R (asseet) (8e,, 81 }]]
sup DED
E (81181 (1) M
[D(s, S1+1)] Bissive 1nmtr
[D(s Sc+1]] Tuuee
DARE
+ sup DED
E (s U (p(s): $1 S141) (( s(t1)l
E Bisnsee SL))))
sup DED
E (11,0141) TpaRe [D S141)l E 1
[D(s Sc+1]]
SDARc
+ W sup DED
E B( (D s(+1)l Be (D( (((
According to McDiarmid 's inequality, with probability at least 1 2. the following inequality holds
B(ee Prr [D 86+1)l (( P [116))
sup DED
DABC
log(4/0)
< E
sup DED
B 1 dortumn
[D(s, ((t1)) + 2A
2m
m
1og(4/8)
< 2E. grin rin
sup DED
Dis st)
+ +2
m
2m
log(4/8)
< <2R (m)
+ +2A
"DARC
2m
log(4/5)
22(m)
+ +64
FDARC
2m
By similar derivation, we can have
W sup DED
E (ss (Ds,,
B(onoet) (($s,,(+11)
(4g(4/8)
S 22 (m) 6W GWA
2m
Thus, we have
dp (
Tasuee T(
trg
d
Woaee?
TC
trg
log(4/8)
22 (m) rane
2W (m)
(6W + 1)
2m
Then, based on Theorem 2 in [38, we can conclude that
Bpsc ATARC E((enaa))
Σ Er(oraa))
R pari
< <||r|||| inf df manec =c
+ (m) SDARC
+ 2W (m)) + (6W + 1
log(4/0) 2m
+ +
This completes the proof.
Theorem B.5. Let πt" argmax,, Err.po E,r(si, be the policy maximizing the cumula- tive reward in the target domain and Ç be the policy learned from DARAIL. Let m be the num- ber of the expert demonstration and R (m)
E, [sup > o,D(s,, St=1]] be the em- pirical Rademacher complexity. Let B be the error bound of DARC the source domain, i.e. E Epec Ppan EE, r(5,, ar) + H H(a, st]] Eppne r(s,,,,))) < B and W be the upper bound
17

---

of the importance weight, i.e. ((41, < W, V(S,, Let discriminator class 'D be A L-bounded function, i.e. [Dw(st: St+1)) I given any (51,8t41) St+1). ||ilp measures the rich- ness of the discriminator to represent the ground truth reward as defined in Appendix B.22 dp defined neural network distance between the (8t, 81+1) distributions generated by the dARcC and "e defined in Appendix B.I. Given the empirical training error of the imitation learning, i.e. Timac:
dp( (frmanc (6
a0 inf (f
F(() SEVOE (0, 1), with probability at least 1 6, we have
r(st,
Σ Er(sias))
- Pooee
Epe t baRc Σ rr(se,a an) + Hlaalee [sc]
E NTMac
Σ ar(suaa
DARC ERROR BOUND IN SOURCE
+ 2W (6W + 1)A loy(4/5)
+ ||llo é + inf dol f 1BB
22 (m))
2m
APPROXIMATION ERROR
ESTIMATION ERROR
IMITATION LEARNING ERROR BOUND
Proof. We can first decompose it into three terms:
Σ Er(or,aa)
Σ Er(or,uu ar)
B
p
Σ ((s,,
Epsos
E((s + H((rls))
E Pacc
Σ r(se, at) + H[ael0a)]
Eep FBARC
Σ a())
E Purt TDARC
Σ Er(se,at) at)
E p
Σ r(ss,ai))
Based on the formulation, ADARC generate optimal trajectories for the target domain in the source domain so that 7 = O. Also, the 12 term is the training error of the DARC and the entropy term of the optimal DARC policy, and only need to bound the 13 terms. Combining Lemma B.4, we have
assume together they are bounded by B. Then,
Σ r(se,aa)
Er(seaa)
E Ex
< Epacstiancc Er(se ar) + H[arlsc]
E STARC
Σ Er(so,a az)
DARC ERROR BOUND IN SOURCE
+ || + inl d (f
Lt TE
27 (m2)
2WR + (6W + 1)
log(4/8)
DARC
2m
APPROXIMATION ERROR
ESTIMATION ERROR
IMITATION LEARNING ERROR BOUND
18

---

C
Additional Experimental Details and Results
Code is available https//wiihloooooooooooooo/oo/ot
Comain-Adaptationn -nd Sedewardd-Agmmenteee Imitattion
C.1 Estimation of Ar(st, at, 8t+1) and importance weight ( (ec (1(er,a))
Following the DARC Bl, the importance weight can be estimated with the following two binary classifiers p(trg St, an) and p(trg $t, at, St+1) with Bayes' rules:
p(rg se, ae St+1) Pug ( [s, (ppppppppsss) st. a (ggg)))((ggg//g /ppp on stt1),
(C.1)
p(8,, 0z (rg) p(rgp st, ae (St pptgg)
(C.2)
Replacing the p(s,, at trg) in Eq.
C.1 with Eq. CC.2)
obtain:
p(trgl Se, Cr 814 ()()))((s,,s a, Sc+1)
Pug ( se, ar)
p(trg s;, at (p()p
Similarly, we can obtain the Pere (81+1 (8,, ae)
We calculate the Ar(se, Oe; 8141) following:
Pug Pug( Stc Qe)
p(si, S6+1) = log
Parc ( 18,, ax)
log p(rgg) 8t, at, 81+1) - log ((tgg/st, 0x) + log p(re(s)) 8141) log az).
p(8 8141) can be obtained from p(8,, = exp [Ar(se, de, 8t+1)]
Training the classifier p(trgi ax) and p(trg| St41) The two classifiers are parameterized bu OSA and OSAS. To update the classifiers, we sample one mini-batch of data from the source replay buffer D and the target replay buffer Då respectively. Imbalanced data is considered here as each time we sample the same amount of data from the source and target domain buffer. Then, the parameters are learned by minimizing the standard cross-entropy loss:
LSAS = DE [log Posss (trgl St, Oe, St+1)] EDp [log Posas (trglst, at, 8++1)]
CSA DE [log Pal. (trgl 8t, ag Se+ }] Epg [log Pas (trg St. aes S24 )
Thus, (ASAS, ASA) obtained from:
argmin LCE Dig
argginn LsAS + LsA
C.2 Description of Baseline Methods
Importance Sampling for Reward (IS-R) With (8t, at, 8t+1) from the source domain, the IS-R directly re-weight the reward in each transition. We can view IS-R as learning the SAC with rewards (ag (as,as)
(e(Se, at) and seeking to maximize the following objective:
Paols a 188.9
Σ Purg (Se+ 18t, as) Pere 18411188 re re( at)
max T
(s. (1 (e)
Importance Sampling for SAC Actor and Critic Loss (IS-ACL) Another way of doing importance sampling by re-weighting the actor and critic loss SAC. The loss for the Q-network in SAC becomes:
Pug Pug (uc($141|se. (Qo(se, ar) - 2(3,, ae, d))²
min E Ess, (-ss) (p
Pere (Se.4 18,, ax)
19

---

where d is the done signal, and the target is given by:
y(5 ae - 7(1 - d)
min Qug, (s a') 1=1,2
log (cd)s(()))) a' ~ x
The actor loss is:
Par (ag(81+1 at) Perel (st11 (8c, ai) [Q= ((|s))
max Eust
DAIL In DARAIL, the policy is optimized with the reward estimator RAE with the true reward from the source domain. We also want to compare the vanilla imitation learning with importance weight. The objective is:
min max
{ (eporis
E((s), (u(s,, S142) (042) (eee Jor
Σ log(1 (u Stn))))
}
TDARC
(C.3)
Then, following the Eq.(C.. the objective of policy optimization without the reward estimator is:
max Epac.
Σ
p(s 1t+1) log (s (st,,)t+1)
(C.4)
We can view it as reduced version of our proposed method, which uses the reward function provided by the discriminator and importance weight.
MBPO 19). MBPO is model-based RL method. We train the MBPO in the source domain and deploy it to the target domain.
MATL 20). MATL modified the reward on both the source and target domains and aligned the trajectories on both domains. Unlike our method, they need access to the reward from the target domain.
GARAT 10 GARAT a grounded action transformation approach that simulates target transitions (st, Qz S1+1,7) r) in the source domain with modified action, where the modified action is learned from imitation learning.
C.3 Broken with probability Pf
As discussed, we use the broken with probability for Ant and Walker2d. The dynamics shift created by freezing one action varies across environments. For instance, in the Ant robot, the 0-index controls the rotor between the torso and front left hip, while in the HalfCheetah, the 0-index controls the back thigh rotor. So, the broken Ant experiences larger shift than the broken HalfCheetah we break the O-index for both environments. Also, the broken environment in Walker2d and Ant creates such large dynamics shift that overly difficult to adapt from the source domain, i.e., DARC cannot obtain the optimal reward in the source domain. We then introduce the broken with probability Pş to better control the magnitude of dynamics shift. Broken with probubility Pf means the 0-index action is frozen with probability py and follows the commanded torque with probability 1 py- In Reacher and HalfCheetah, the source environment broken with probability 1. Ant and Walker2d's 's source domain is broken with probability of 0.8.
Figure 5 shows the performance of DARC in Ant and Walker2d under different broken probability Ps in the source domain. We can observe that when py 1.0, the performance degradation of evaluating in the target domain is larger than the p 0.8 case. Also, when pf 1.0, the DARC evaluation performance in the target domain is close to O. Moreover, we notice that in the p, = 1.0 case, DARC training performance the source domain receives a much lower reward than the ps = 0.8 case. However, we want to mimic the DARC behavior in the imitation learning, so want DARC to be able to receive optimal reward in the source domain. Thus, for the Ant and Walker2d environment, we choose py = 0.8 for the source domain.
20

---

Tahning Roward
4000
Rariig Reward
1005
Trair Roward
Reward
4020
Eloluttion Roward
Opoma Reward barget
3000
Traiinng Rewand
3000.
Optimal Rewardd Darget
Eearaciinn Reward
Oatimal Rewand Taget
Evatiition Reward
2000
Evaluation Reward
Opramo Reward baree
2000
1800
Tainng
Fairing
(a) Ant, Pf 0.8
(b) Ant, Pf =1 (c) Walker2d.p5 0.8 (d) Walker2d, Pf
Figure 5: Training reward in the source domain, i.e. EmpaRC E, r(st ae)], and evaluation reward in the target domain, i.c. Bruua |E, r(st, at], for DARC in Ant and Walker2d with different broken probability Pf in the source domain. (a) and (c) shows the performance of DARC under DARC under py 1.0. The performance of DARC under Pf = 1.0 is much worse than the case Pj 0.8, and the performance gap between DARC in the source and target larger, showing that the dynamics shift is overly large to adapt and learn good expert demonstration.
Pf 0.8, and (a) and (c) shows the performance
C.4 Training Curve of the DARAIL and Baselines
We show the training curve DARAIL and baselines in different environments under the broken source environment setting in Figure 6 corresponding to the result in Table 3 We also show the training curve modifying the configuration in Figure and
HaifCheetah
kerdd
2006
-Db-
-00
-190
100 AN T3
X1 Training Staps
1270
1900
Training
Trining Stops
Training Stops
DARC Evaluation
DARAIL
MIPOO
GARAT
Training
Target
Figure 6: Upper horizon line: DARC reward the source domain. Lower horizon line: DARC reward in the target domain. The figures show the mean value of multiple runs and the standard deviation. The figure shows that our proposed method performs better than DARC in the target domain and other baseline methods.
Table 5: Comparison of DARAIL with DARC, 0.5 gravity.
DARC Evaluation DARC Training Optimal in Target
DARAIL
HalfCheetah Ant Walker2d
1686 = 392
5721 + 463
7559 + 782
5485 + 592
2058 + 553
348 + 71
3380 + 538
990 + 12
706 = 64
936 + 158
2830 + 482
878 # 122
Reacher
-13 + 1.3
-11 +
-7.2 + 0.3
-12.2 + 0.5
21

---

1007
Traning Steps
Trining Steps
Faining Steps
Thaining Sieps
02
Training Steps
1o Steps
Training 22
Triiiing
DARC Evalutioo
DARAIL
I5R
MBPO
GARAT
DARC Training
DAIL
15-A
MATI
Optimal Reward in Target
Figure 7: Training Curve of changing gravity setting. Top: target domain gravityx0.5, button: target domain gravityx11.5 Upper horizon line: DARC reward in the source domain. Lower horizon line: DARC reward in the target domain. The figures show the mean value of multiple runs and the standard deviation. The figure shows that our proposed method performs better than DARC in the target domain and other baseline methods.
HellCheetah
Art
Welkerdd
Beacher
18000
Steps
Tniring Steps.
1839
300
300 03
clc 5.5 28
L3o 13
13
20
25
Training 26
2000 1a.0
Steps
Training
Training Steps
DARC Evaluation
DARAIL
R
MBPO
GARAT
Training
DALL
I5
Optimal Reward
Figure& 8: Training Curve of changing density setting. Top: target domain density x0.5, button: target domain densityx 1.55. 1.5.
Upper horizon line: DARC reward in the source domain. Lower horizon line: DARC reward in the target domain. The figures show the mean value of multiple runs and the standard deviation. The figure shows that our proposed method performs better than DARC in the target domain and other baseline methods.
8000
4200
4000-
Toriing Roward
3000
Triniig Ronord
4000
Tow sacion Reward
Exoluation Roward
2002
Evalcation Ranard
Thi ning Researd
Opémal Reward Target
1500
Optinol Reward Torget
40
Tathing Ralann
Eostastion Deward
Optiraal Ranoard Target
Optimal Roward in Tarpet
20000
Traising
Thiining
Trairing
(a) HalfCheetah
(b) Ant
(c) Walker2d
(d) Reacher
Figure 9: Training reward in the source domain, i.e. E E Σn r(st az)), and evaluation reward in the target domain, i.e. E [Σ, r(st, a,)], for DARC in four environments. Deploying trained DARC policy to the target domain will cause performance degradation.
22

---

Table 6: Comparison of DARAIL with baselines in off-dynamics RL, 0.5 gravity.
DAIL
IS-R
IS-ACL
MBPO
MATL
GARAT
DARAIL
HalfChcetah
3671 = 331
3432 + 332
4896
249
12.2 42
741 + 195
3436 = 226
4093 + 1021
Ant
970 1: 16
982 3.6
984 + 77
981 3:
980 1 46
976 105
990 + 12
Walker2d
541 315
3:35
1267
793
721
767 5:16
823 458
122
Reacher
-12.5
-82 2.6
-7.1
2.6
-16.2
13.6 0.1
-13.7
-1.2 : 0.5
Table 7: Comparison of DARAIL with DARC, 0,5 density.
DARC Evaluation DARC Training Optimal Target
DARAIL.
HalfCheetah
8328 + 861
8790 + 486
9970 + 983
10308 + 1042
Ant
1587 + 224
2170 + 195
3798 + 341
3472 + 245
Walker2d
773 7
2449 + 234
2729 + 492
1595 + 168
Reacher
-13.3 1.2
-9.4 1.5
+.2
-12.2 11
Table 8: Comparison of DARAIL with baselines in off-dynamics RL, 0.5 density.
DAIL
IS-R
IS-ACI.
MBPO
MATL
GARAT
DARAIL
HalfCheetah
4433 1 453
5332 1 1063
6951 + 1067
740 + 172
2676 1 315 2437 213
10308 1042
Ant
2233 1 809
2050 1: 892
2396 1 96
980 -: 102
1961 1 611
2149 1: 406
3472 1 245
Walker2d
1930 + 441
646 226
1180=789
391 191
441
480
1595 + 168
Reacher
-12.2 1.8
-13.3
113.2=1
-11.7
1..
-14.1
=1
Table 9: Comparison of DARAIL with DARC, 1.5 density.
DARC Evaluation DARC Training Optimal
DARAIL.
HalfCheetah Ant
8833 = 539
9380 : 728
6309
11515 = 335
5961 970
6036 + 1345
3288
5193 + 463
Walker2d
760 430
3288 + 849
3383
2674 + 376
Reacher
-10.4 + 0.4
-
-7.1
-10.2
Table 10: Comparison of DARAIL with baselines in off-dynamics RL, 1.5 density.
DAIL
IS-R
IS-ACL
MBPO
MATL
GARAT
DARAIL
HalfCheetah
5057 + 766
4814 + 524
4966 + 727 3598 + 706 530
320
3650 + 875
11515 + 335
Ant
2738 + 781
3335 + 1010
3499 + 967
2371 + 604
3135 + 463
3028 + 690
5193 + 463
Walker2d
997 432
1452 + 1036
1950
198
448
1498
176
1066 +: 739
2674 +: 376
Reacher
-11.3 1.0
-15.2
1.0412.0
-14311
-11.12
-10.2 1.3
2.1
23

---

DARNI
10a0
DMKC Bro
DMRC Taaing
2303
Optimal Rewoed Target
DABALL
DARAIL
BANAIL
DARC Duslustion
DARC Eiotation
DARC Evalustion
DARC Toining
DARC Traring
DARC Taiking
Optimal Resord beget
Optimal Renard n Targe:
Optimal Raward in Target
Training
Training Staps
Training Steps
222 ese solos
(a) HalfCheetah
(b) Ant
(c) Walker2d
(d) Reacher
Figure 10: Experiments of DARC and DARAIL the intact source and broken target setting. We observe that the DARC does not have significant performance degradation. Also, we show that DARAIL can perform similarly to DARC in this setting.
C.5 DARC training and evaluation performance on broken source setting
Figure shows the performance DARC trained the source and evaluated in the target domain under broken source environment setting, The training reward is the reward obtained in the source domain, i.e. ETTRACC per [E, r(st, az)] and the evaluation is the reward deployed in the target domain, i.e. [EADARC,PY 1E, ( d.)]. We observe the performance degradation the figure 9 Empirically, we notice that the DARC policy performance in the source domain, B E r(st. at)]. is close to the optimal reward in the target domain which matches with the DARC objective that DARC can generate target optimal trajectories in the source domain. However, deploying it to the target domain will result in performance degradation and suboptimal reward due the dynamics shift.
C.6 Performance DARAIL on broken target environment
We show the performance of DARAIL in the intact source and broken target environment setting Figure (the (the setting in DARC paper (3). We observe that our method outperforms the DARC reward in the target domain, E pm [ r(se, dt)]. Also, we see that the performance of DARC in the source domain and target domain are very similar. Compared with the performance gap when the source environment is broken in Figure ! As discussed, DARC works well when the assumption that the target optimal policy performs well in the source domain is satisfied. In the broken target setting, the target optimal policy can perform the same the source domain.
Further, empirically, in the broken target setting, the DARC policy learns near O value for the broken joint, which guarantees that the policy can generate similar trajectories in the two domains. Also, maximizing the adjusted cumulative reward in the source domain with policy with a near O value for the broken joint equivalent to maximizing the cumulative reward in the target domain. Thus, DARC perfectly suits the broken target setting. However, in the broken source setting and other more general dynamies shift cases, the target optimal policy might not perform well the source domain. For example, in the broken source setting, the target optimal policy will perform poorly in the source domain as it loses one joint the source domain. Another way to understand why DARC fails is that it learns arbitrary value for the broken joint, which becomes detrimental in the target domain. However, this is just an artifact of the particular setting. As we discussed above, the intrinsic reason that DARC fails is the violation of the assumption.
C.7 Performance of mimicking source optimal trajectories
In Figure I1 We compare DARAIL, which uses DARC trajectories in the source domain as expert demonstrations and mimicking source optimal trajectories regardless of the target domain.
C.8 Access to the target domain data compared to DARC.
Both DARC and DARAIL require some limited access to the target rollouts. DARAIL, the imitation leamning step only rolls out data from the target domain every 100 steps of the source domain rollouts, which 1% of the source domain rollouts. We claim that more target domain rollouts will not improve DARC's performance due to its suboptimality. and DARAIL is better not because of having more target domain rollouts. We verify by comparing DARC and DARAIL. with the same amount of rollouts from the target domain in the broken source environment setting

---

DARAL
DARNIL
Oponall Reward Source Optiaal Reward Target
Optinal Reward Source
295
Optinal Reward Brrget
2001
DiRaR
Optimal Roward Source
Optimal Roward in Target
63
Training Staps
Training Steps
1953 sos
K00
Opinall muaes
DARAL
1030
Optimal BeAnnd Source
200
Optimal Reward Target
1000
200
Optinal Reward Sourcc
DARAIL
Outinal Reward Source Optinal Reward Targer
DARAIL
-209
Optinal Reward Target
35
Training Stops
Training Steps
(a) HalfCheetah
(b) Ant
(c) Walker2d
(d) Reacher
Figure 11: Experiments on using source optimal policy as the expert demonstration instead of the DARC policy as the expert demonstration. Mimicking the source optimal trajectories will not receive similar performance mimicking DARC performance, and there is big performance gap between the source optimal reward and imitation learning performance in the target domain.
in Tables 1T and 12 Specifically, we examine DARAIL with 5e4 target rollouts alongside DARC with 2e4 and 5e4 target rollouts. DARAIL has 5e3 target rollouts for the Reacher environment, while DARC has 3e3 and 5e3 rollouts. From the results,
that increasing the target rollouts from 2e4 to 5e4 (or from 3e3 to 5e3 in the case of Reacher) does not yield a significant improvement in DARC's performance due to its inherent suboptimality. Notably, DARAIL consistently outperforms DARC when given comparable levels of target rollouts.
Table Comparison with DARC with the same amount of rollout from the target. The number in the columns represents the amount of rollout from the target. More target domain rollout will not improve the DARC's performance further. Experiment on broken source setting.
DARAIL Se4 DARC Evaluation 2e4 DARC Training 2e4 DARC Evaluation Se4 DARC Training 5e4
HalfCheetah/
7067 = 176
4133 + 828
6995 + 30
4037 + 798
6988 + 27
4752 872
4280 33
5197 155
4342 +
5207 172 3782 = 510
Walker2d
4366 - 434
2669 + 788
3896 + 523
2538 + 802
Table 12: Comparison with DARC with the same amount of rollout from target, Reacher. The number in the columns represents the amount of rollout from the target. More target domain rollout will not improve the DARC's performance further. Experiment on broken source setting.
DARAIL 5e3 DARC Evaluation 3e3 DARC Training 3e3 DARC Evaluation 5e3 DARC Training 5e3
Reacher -13.7 + 0.9
-26.3 + 3.3
-11.2 + 2.9
-29.7 + 4.1
-10.2 + 1.2
25

---

HalfCheetah
Walker2d
5000
Rewards
Rewards
3000
4000
2000
2000
1000
0
0
-2000
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Training Steps
Training Steps
DARC Evaluation
=
1
n 10
50
DARC Training
Figure 12: Experiment on how cumulative n-step importance weight performs on DARAIL. Per-step importance weight significantly outperforms using the last n-step multiplication of the importance weight.
HalfCheetah
Walker2d
7000
3000
5000
Rewards
Rewards
2000
3000
1000
1000
0
11000
0.0
1.0
0.5
3.0
Training Steps
Training Steps
DARC Evaluation
n 10
50
DARC Training
Experiment on how cumulative n-step importance weight performs on IS-R in broken source setting. Per-step importance weight significantly outperforms using the last n-step multiplica- tion of the importance weight.
Figure 13:
D Ablation Study
D.1 Per-Step Importance Weight Cumulative Importance weight
In our paper, to reduce the variance, we use the per-step importance weight Pr((s,are1) for the importance sampling method and DARAIL. Here, compare the per-step importance weight with the cumulative n-step importance weight, which is the multiplication the weight before time step t:
(sc(s1
Pug Pug ( 11441/100 as)
Palst, St+1)
II
Pate ( 18;5 aq)
Note that here, the importance weight the multiplication of the last n steps weight instead of the multiplication from = 0 to i = t. Because the cumulative importance weight might have a NaN value due the product. Thus, the optimization step for the imitation learning of DARAIL as follows:
Σ pm (St, Set ( 8841) (1 pa(Se, 81+1)) log Dwo sttt)].
max Paek
26

---

Similarly, the objective of IS-R 13-Riz is:
max E
Pa Stt1)r 8441)
Pun
We compare the per-step importance weight and the cumulative n-step importance weight on DARAIL and IS-R. Specifically, we consider [10. 50] for HalfCheetah and Walker2d, respectively. We show the results of DARAIL in Figure 12 and the results of IS-R in Figure 13 We see that the cumulative importance weight doesn t perform well on both methods and environments. In HalfCheetah, we can observe that the 10-step cumulative importance weight performs better than the 50-step one. And similar patterns appear the Walker2d. Thus, we can conclude that per-step importance weight will have lower variance and be more favorable in our experiment.
D.2 Update Steps of Discriminator
5000
DARC Evaluation
DARC Training
-20
updale steps
update steps
DARC Trainin
Evaluation
update steps 500
update steps 1000
+40
DARC Trairing
Evalaation
update steps 10
update steas 50
steps
1000
-40
update steps 500
update steps 1000
update steps 1000
-70
update sleps 2000
15
155
20
2%
10000
2000
30000
40000
Training Steps
Training Steps
Training Steps
(a) HalfCheetah
(b) Walker2d
(c) Reacher
Figure 14: Experiment on the performance of DARAIL under different update steps of the discrimi- nator in broken source setting.
In imitation learning, we alternatively update the generator and discriminator. In practice, we normally update the generator several steps and then update the discriminator once. The update steps, updating the discriminator every how many training steps, is a hyperparameter and is important in GAN training. The smaller the update steps are, the higher the update frequencies are. We tune the update steps and show the result of it in different environments. The best discriminator update step in HaifCheetah, Walker2d, and Reacher are 50, 50, and 1000, respectively. We varied the discriminator update steps in HalfCheetah and Walker2d in [10, 50, 500, 1000] steps, and the update steps in Reacher are [50, 100, 1000, 2000 steps. Figure 14 shows the effects of different discriminator update steps in the final performance. As we sec, for all three environments, smaller update step (higher update frequency) is preferred as it can learn better reward estimation. However, as we noticed, for example, for HalfCheetah and Walker2d, when the update step is 50, decreasing it to 10 will not further improve the performance.
D.3 Increase the weight on the modified reward of DARC.
We tested DARC algorithm with modified reward r(st, az) + nA(sE, de, 81+1) with 7) instead of 1. And the n 1 is derived from the distribution matching objective in (333). We show the results in Figure 15 under the broken source environment setting. We can see that increasing 1) will not increase the DARC performance in the target domain but will hurt the performance of DARC in the target domain.
D.4 Hyperparameters
For fair comparison, we tune the parameters of baselines and our method. The hidden layers of the policy and value network are [256,256] for the HalfCheetah, Ant, and Walker2d and [64,64 for Reacher. And the hidden layer of the two classifiers is [64] for the HalfCheetah, Ant, and Walker2d and [32] for Reacher. The batch size set to 256. We regularize the state by adding
running
average of the state. We fairly tune the learning rate from [3e 4, le 4, 5e 5, le 5). For those methods that require the importance weight A. we tune the update steps of the two classifiers trained
to obtain the importance weight from [10, 50, 100). We also add Gaussian noise N NO, 1) to the
27

---

0000
4000
1020
Trairing Roward
3000
Tasning Farkard
5000
Evaluation Reward
Exstustion Beward
2003
Evaluat.on on Raward
Thiriing Rewardd
Octimal Reward Taeget
1000
..0 Optima Roward in Targer
Traniing Revaed
Evaluttin Reward
Op0 Raward Target
Optimnal Rovard in Target
20000
40000
Training Stops
loining
Taiiing Stops
0200
DARIC Rainng
DAHI lonnng
DARC Exoluation
GAMC Extluttoon
Optimal Roward Droct
Taining
Optimal Reward Targe
DARC ninning
2000
4000
Dostaatio
DARC
brget
Optinnal Roward brget
20050 Stens
Trairing Stops
Taining Stess
Trining
1000
6300
60000
2000
-20-
Taining
DAIC Thaning
5000
2000
Turget
Optimal Reward larget
DARC Thaniing
Training Shops
Traiinng Stepe
Trahing Staps
Taiinng Stops
(a) HalfCheetah
(b) Ant
(c) Walker2d
(d) Reacher
Figure 15: Experiment of different in the modified reward r(s az) +7 at 8t+1) for DARC in broken source environment setting. Top row: 1, middle 17 1.5 and button 7) 2. We observe that increasing the will reduce the performance degradation in most cases, but will also harm the performance of DARC in the target domain increasing 7 focuses the DARC perform more similarly in both domains instead maximizing the cumulative reward.
making
input of the classifiers for regularization, and the noise scale is selected from [0.1, 0.2, 1.0]. For the imitation learning component, the number of expert trajectories is 20. We further tune the update steps of the discriminator and add Gaussian noise to the input of the discriminator.
D.5 Computation Resources
We run the experiment on a single GPU: NVIDIA RTX A5000-24564MiB with 8-CPUs: AMD Ryzen Threadripper 3960X 24-Core. Each experiment requires 12GB RAM and require 20GB available disk space for storage of the data.
E
Limitations
A potential limitation will be that rely on DARC similar methods generate state pairs. An overly large dynamics shift, or data limitation may prevent us from obtaining high-quality state space data to imitate in the source domain. We do the experiment the Mujoco environment instead of the real-world sim-2-real problem. We leave the investigation of this to future work.
28

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and Introduction section states the contribution. And in the introduc- tion section, we have a contribution list.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We talk about the limitation of our method in the Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
Yes. We present our theoretical result in Section 4 and the proof is in Appendix
B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details about how we create the dynamics shift and the hyperpa- rameters that used in the experiments in Appendix D.4 and release the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
30

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide GitHub repository in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc.
hhbllic/gulless/odeeebbmm.ssss httssSubmmmsiiooocoooo.oo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ccpppi.ccppulli///grg guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide the details of the experiment setting, including how to create the dynamics shift the Experiment section. We also describe the hyperparameter tuning in the Appendix D.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have multiple runs of each experiment and report the mean value and standard deviation in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the GPU/CPU as well as the RAM and storage information for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more computing than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /
Answer: [Yes]
Justification: Our data open source benchmarks in the RL research field.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the conclusion, we briefly mentioned that our method avoids directly training policy in a high-risk environment in safety-critical tasks.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
32

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We run the experiment on the simulated RL benchmarks; thus, no such issue
exists.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We provide citations to all the data and related work in our paper.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include the code in our paper. Also, details about the implementation included in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our experiments are conducted on the RL benchmarks and thus do not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research and experiment don't require IRB as conducted experiments on simulated RI. benchmarks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "nS5mWxSjFq", "reviewer": "Reviewer_mGy8", "summary": "The authors introduced the approach Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) for off-dynamics RL. The work aim at generating the same trajectories in the target domain as expert trajectories learned via DARC in the source domain. With GAIL-style framework and reward augmentation, DARAIL reached SOTA in bench mark off-dynamiacs environments. DRAIL does not assume the learned expert policy in the source domain to be close to the optimal policy in the target domain, and the authors are able to give a theoretical error bound of the method according to the dynamics shift scale.", "questions": "How would DARC trajectories quality influence the overall imitation learning performance in the target domain?", "limitations": "Did not test on more complicated dynamics shifts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The problem of off-dynamics RL is very meaningful, given some target environments and data are not easily accessible.\n2. The shortcoming of DARC is clearly stated. Movitation is clear.\n3. The theoretical bound proof is neat.\n4. Sufficient experiments on both on baseline comparison, sensitivity analysis and dynamics shift sclaes.", "weaknesses": "1. Typo in Line #81, expectation should be over p_trg\n2. Typo in Appendix C.2 Line #537 (C.1)\n3. Typo in Appendix C.2 Line #540, #546\n4. There is no comparison experiements of directly using (s,s’) from the expert trejectories in the source domain for DARAIL.\n5. Freezing the 0-index action is not a good way to validate the effectiveness of the method on dynamics shift. Experiemnts on ,ore diverse dynamics shift environments are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank the reviewer for their time and comments. We now address your concerns point by point. \n\n---\n> How would DARC trajectory quality influence the overall imitation learning performance in the target domain?\n\nWe use trajectories generated by the DARC in the source domain as the expert trajectories for imitation learning, which resemble the optimal trajectories in the target domain, according to the DARC’s objective. In general, the imitation learning performance should be similar to the DARC performance in the source domain, with the latter one expected to perform similarly with the target optimal policy if the DARC learning error is small. The better the DARC trajectory quality, the better performance the resulting policy from imitation learning can achieve. Overall, our analysis in section 4 shows that a good DARC trajectory quality will have a smaller learning error in the first term of the error bound.  Further, we want to emphasize that our method is not particularly limited by the performance of DARC because our method can be viewed as a general framework that uses imitation learning to transfer trajectories from the source domain to the target domain as long as we can obtain trajectories that resemble the optimal trajectories on the target. \n\n---\n> There are no comparison experiments of directly using $(s,s’)$ from the expert trajectories in the source domain for DARAIL.\n\nWe assume that by “$(s, s’)$ from the expert trajectory in source,” the reviewer means using the optimal trajectory in the source. We provide such experiment results on HalfCheetah and Reacher in Table 2 and 3 in the attached PDF. We first train a policy in the source domain (do not consider the dynamics shift) and use imitation learning to transfer it to the target domain. The results show that DARAIL outperforms the method that uses optimal trajectories in the source domain as the expert policy for imitation learning because DARC can generate trajectories that resemble the optimal ones in the target domain.\n\n\n---\n> Experiments on other off-dynamics shifts.\n\nWe provide additional experiment results on HalfCheetah and Reacher in Table 1 in the provided PDF. The off-dynamics shift is constructed by modifying MuJoCo's configuration files. We modify the coefficient of gravity from 1.0 to 0.5 for the target domain. Our results show the effectiveness of our method. In our revision, we will add more experiments on these more general cases to demonstrate the performance of DARAIL.\n \nWe thank the reviewer for their detailed suggestions for improving the readability of our paper. We will follow them closely, fix all the typos, and clarify the writing of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "workrblLCC", "reviewer": "Reviewer_LdFt", "summary": "The paper considers the off-dynamic RL setting and identifies that existing approach, DARC, fails to obtain the true optimal policy in the target MDP. Leveraging ideas from imitation learning literature, the paper proposes a learning objective that takes into the account the dynamic shift between the source and target MDPs. The objective is theoretically sound and the conducted experiments show that the proposed method, DARAIL, is able to account for severe dynamic shift in MuJoCo environments.", "questions": "**Questions**\n- On page 4, lines 120-122: Is this always true, or just may? If I understand correctly since the dynamics are \"unknown\" it may recover the wrong target policy (i.e. $\\pi^* \\neq \\pi^{trg}$)?\n- DARC does not need any online learning while DARAIL does correct? I believe this should be highlighted if that is the case.\n- On page 4, lines 151-152: This shared reward is an assumption correct? What if this is not true either?\n\n**Possible typos**\n- Page 2, line 128: Domain", "limitations": "**Limitations**\n- The experiments are purely in MuJoCo so I am curious what can happen in other domain.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The experiments show that DARAIL is able to account for severe dynamic shift and is not too sensitive with clipped importance sampling ratio.\n- The relationship between the proposed estimator and doubly robust estimator is interesting.", "weaknesses": "- The notation can be defined more clearly.\n\t- On page 3, line 106: Is $\\tau$ a trajectory sequence of state-action pairs, or just sequence of states?\n\t\t- What generates this?\n\t- On page 3, Eq (3.2): Is $\\tau_{\\pi^*}^{trg}$ Is this trajectory generated by $\\pi^*$ in the target domain?\n- It will be preferable if the paper can provide explanations on why DARC fails to address for the dynamic shift.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank the reviewer for their time and comments. We now address your questions point by point.\n\n1, **Notation of $\\tau$.** $\\tau$ represents the trajectory sequence of state-action pairs, so it is a notation for the trajectories. For example, we use $\\tau_{\\pi_{\\theta}}^{\\text{src}}$  to represent the trajectories generated by the policy $\\pi_{\\theta}$ in the source domain. \n\n2, **Why DARC fails to address dynamics shift.** First, as mentioned in the general comment, DARC works well under some dynamic shifts when the assumption about the dynamic shift made in the DARC paper is satisfied, i.e. the optimal policy for the target domain is also a good policy in the source domain. DARC works well in this setting because the objective of DARC can also be viewed as maximizing the cumulative reward in the source domain under a constraint that the learned policy performs similarly in two domains. However, in more general off-dynamics settings. This assumption is not satisfied. DARC will be suboptimal in the target domain as DARC does not perform similarly in the two domains, leading to performance degradation in the target domain. \n\nOur method is motivated by DARC, which generates trajectories in the source domain that resemble the optimal ones in the target domain. We propose to use imitation learning to transfer DARC trajectories to the target domain so that deploying the policy to the target domain will not receive such performance degradation. \n\n3, **On page 4, lines 120-122, recover the wrong target policy.** \nWe would appreciate any clarifications on the definition of $\\pi_{trg}$ in the second part of the question. To answer the first part, the matching of the distribution is the main goal of the DARC’s learning objective. Under the unknown dynamics and unknown dynamic shift, DARC tries to learn a policy that generates trajectory distribution similar to the target trajectories from the target optimal policy. But the policy $\\pi_{DARC}$ will not be the same as the target optimal policy due to the dynamics shift. Here, we only use the $\\pi^*$, which is the target optimal policy, in our analysis of DARC’s failure. \n\n4, **DARC does not need any online learning, while DARAIL does.** DARC does need online learning and they sample $(s_t,s_{t+1})$ from the target domain. Similarly, our method requires sampling from the target domain. In both cases, the amount of data rollouts from the target is significantly smaller than the rollouts from the source. The details can be referred to the third bullet point of the general response. \n\n5, **Assumption about the shared reward**. In the off-dynamic RL setting, we specifically focus on the setting where the two domains only differ in the dynamics but share reward functions and state/action space. If this assumption is not true, this becomes a different domain adaptation problem, and we might need other techniques to adapt to the shift in the reward function. \n\n6, **Experiment on other settings instead of Mujoco.** We follow the recent work in off-dynamics RL and experiment on the MuJoCo environment, which is easy to access and analyze. We will also add new experiments on other environment settings in our revision.\n\n\nWe appreciate the reviewer's feedback. We hope we have thoroughly addressed your concerns. We are willing to answer any additional questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XyVGAMnJcf", "reviewer": "Reviewer_VFUe", "summary": "This paper provides solution to a specific kind of domain adaptation, where the source action space is a subset of the target action space. In this setup, the source policy might fail in the target domain, because the policy would output arbitrary values for the action dimensions only active in the target domain, but not in source. To resolve this, the paper disentangles the effect of actions, and considers the source domain's policy's state trajectories as the optimal trajectories to imitate in the target domain. By applying imitation learning from observation, they can learn a target policy to reproduce the state transitions from the source policy, but using the action space from the target domain. One trivial solution to this is that the target policy would learn the source policy's actions, just with a 0 output (or a value that leads to no effect) for the extra dimensions in the target domain. The paper outperforms the prior work DARC on its specialized environment setup, but unlikely to be a general method for other off-dynamics setups.", "questions": "I have listed suggestions to improve in the weakness section above. There are a lot of changes in the paper presentation that are necessary, but my major concern would still be about the limited applicability of the proposed problem setup and the method not even best suited for that proposed problem.", "limitations": "The paper mentions a couple of limitations in the appendix section, not in the main paper. The paper does not discuss several limitations of their problem setup's applicability, their method's potential fallacies on the original DARC environments, and their limiting assumption of source trajectories being optimal for the target domain.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper provides a good overview of the prior work DARC, and helped to understand their method without having to go through DARC.\n- The experimental evaluation consists of several baselines, and DARAIL generally outperforms them, achieving similar to source performance. This shows that the imitation learning from observation pipeline is working.\n- They provide code for reproducibility.", "weaknesses": "## Evaluation on a very specific kind of dynamics change\nThe major issue with this work is its limited evaluation where the only kind of off-dynamics shift is freezing the 0-index value to 0 in the action in the source domain. Ideally, if the action support of source is a subset of the target domain, then the same policy should be perfectly fine in the target domain. Yet, this paper shows that the same policy underperforms in the target domain. The only reason why this might be happening is because the DARC policy learns an arbitrary value for the 0-th dimension of action torque, which is inconsequential in the source domain, however, becomes detrimental in the target domain. The paper never discusses this obvious reason for this failure, and the only thing a method needs to do is to learn to always output 0 for the 0th-index, for it to work in the target domain.\n\nDARAIL is specifically constructed so that it learns to output 0 and works well for this artificial environment setup, because it **assumes the source trajectory to be the optimal trajectory in the target domain**. But this is a major assumption not made in the original DARC paper. In fact, I would argue (and I am open to change my mind on this if proven otherwise) that DARAIL does not work in the off-dynamics settings considered in DARC paper. For instance, consider ant which is crippled in the target domain but not the source domain (from DARC paper). Considering the source ant's trajectories as optimal could be detrimental for the target crippled ant, because most of the state transitions $(s_t, s_{t+1})$ in source domain are not even possible in the target domain. The crippled ant requires a different optimal policy to be learned, which cannot be obtained by just imitating the source state trajectories, which DARAIL would do.\n\nIn order to show that DARAIL is actually a general solution to off-dynamics RL, the paper should include experiments on:\n- All the same setups provided in DARC paper.\n- At least one more example of off-dynamics RL other than just freezing the 0-index of action.\n\n## Paper Presentation\nThe paper is often written in a confusing manner and there are several quality issues in presentation. The following need to be corrected in writing:\n- Figure 6 (b) for Ant and 6 (d) for Walker are the exact same training curves. Please fix this.\n- The paper never discusses why DARC \"fails\" is because of the 0-th action dimension being an arbitrary output in the target domain, because it was never trained to be 0 during training.\n- L2-3: \"performance degradation especially when the source domain is less comprehensive than the target domain\": \"less comprehensive\" is vague.\n- In L23-25, the paper mentions that \"in domains such as medical treatment [1] and autonomous driving [2], we cannot interact with the environment freely as the errors are too costly\". However, in Algorithm 1 L7, it assumes access to the target environment for rolling out.\n- L41-42: \"the source domain has less action support, which is normally a harder off-dynamics RL problem.\" Why is this a harder problem? If anything, target domain having a less action support would mean that the agent needs to adapt to more adverse off-dynamics conditions.\n- L64-65: \"DARAIL relaxes the assumption made in previous works that the optimal policy will receive a similar reward in both domains.\" — What does this mean, because L78 says that both the source and target domains have the same reward function? In fact, DARAIL considers the source trajectories as optimal trajectories for the target domain.\n- What does it mean for the two domains to have the same reward function, when the dynamics are different? Does the paper assume the reward function is of the form R(s) only and not R(s, a, s')? Any assumptions made in the problem setup should be clearly written.\n- In Eq 3.1 and 3.2, the product term $\\Pi_t$ should include the last terms inside the bracket.\n- L110-111: This cannot be right. The optimal policy should be proportional to the exponential of the cumulative returns, not reward at time step t.\n- Figure 1 (a): It is unclear how this figure suggests that there is any **performance degradation**. Since the dynamics are different, it is possible that the optimal reward in the target domain could be just different from the optimal reward in the source domain — comparing these two does not make sense. It only works in this paper because of the artificial formulation that the source action space is a subset of the target action space. The correct comparison would be evaluating $pi_{DARC}$ in the target domain against some $\\pi_{optimal}$ for the target domain, both in the target domain. The training reward obtained in the source domain does not tell us anything for general off-dynamics setups.\n- L151-166: This paragraph is hard to follow.\n- Figure 7: Some baselines are not trained for the same number of steps.\n\n## Method is overly complicated for the problem setup\nThe paper makes a questionable assumption that when $\\pi_{DARC}$ is optimal for source domain, the trajectories it obtains are also optimal in the target domain and thus, we should achieve similar state trajectories in the target domain. While this is a very restricted assumption, let's say for the sake of the argument, this assumption makes sense for certain useful setups. Even then, why learn $\\pi_{DARC}$ using any influence from the target trajectories at all? Why not just learn $\\pi_{src}$ from the src reward only, while disregarding the target trajectories and not applying DARC at all. And then we can transfer that policy using Gaifo to the target environment. What is the idea of doing DARC at all, when anyway the source policy is to going to be used for imitation learning from observations in the target domain? If DARC on src is really important, it should be experimentally validated with convincing arguments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable time and detailed feedback. We hope our response will fully address all of your points. \n\n---\n>Evaluation on a very specific kind of dynamics change.\n\n1) In the general response, we provide additional experiment results under DARC setting and more general off-dynamics settings to show that our method works well beyond the specific dynamics change. In the second bullet, we clarify why DARC fails in the broken source and intact target settings theoretically and empirically.\n\n2)  **DARAIL is specifically constructed so that it learns to output 0.** This is not true. As mentioned in the general response, Table 1 in the provided PDF shows DARAIL works for the general off-dynamics setting other than broken action. Indeed, if we know the difference in dynamics, we can “simply” learn/set 0 for 0th index. However, we do not assume we know beforehand. We will add this discussion to our revision.\n\n3) **DARAIL does not work in the off-dynamics settings considered in DARC.**  It’s not true that most of the state transitions  $(s_t,s_{t+1})$ generated by DARC in the source domain are not possible in the target domain in DARC settings. DARC learns a near 0 value for the 0th index, as mentioned in the second bullet in the general response. Table 3 in the provided PDF shows that DARAIL performs better than the DARC evaluation in the target domain in DARC setting.\n\n---\n>Method is overly complicated for the problem setup.\n\nFirst, **we do not assume that “when $\\pi_{DARC}$ is optimal for the source domain, the trajectories it obtains are optimal in the target domain.**” Instead, we notice that  $\\pi_{DARC}$ generates the trajectories in the source domain that **resemble** the target optimal ones, as this is the DARC’s learning objective. So we transfer $\\pi_{DARC}$ to the target domain instead of mimicking the source optimal policy. We provide additional experiments that mimic the source optimal policy instead of DARC trajectories on two settings in Tables 2 and 3 in the PDF. DARAIL significantly outperforms that, probably because the optimal source domain trajectories might not perform well in the target domain. \n\nSecond, we want to clarify further that we do not assume that the source trajectories generated by DARC are **exactly** the optimal trajectories in the target domain. Instead, we utilize DARC’s learning objective to learn a policy that generates trajectories in the source domain that **resemble** the target optimal ones. Moreover, we considered DARC's training error in the first term of the error bound in our analysis. \n\n---\n>Paper presentation questions\n1) **Performance degradation happens when the source domain is less comprehensive.** We want to clarify that less comprehensive means that the support in the source is less than the target, such as our setting. However, we agree that this is not the only case when performance degradation happens, as suggested in Table 1. We will clarify this point in our revision.\n\n2) **Why is it a harder problem when the source domain has less support?** In the domain adaptation problem, if the source domain/training data has larger support or covers more data distribution than the target, it would be easier to generalize to the target domain because the training data has seen all possible target actions/data. In off-dynamics RL settings, if the target domain has less support, we can restrict the action support in the source domain. For example, in the DARC setting, whose target environment has less support, the DARC policy learns to output near 0 value for the 0th-index, and the problem becomes optimizing the policy in the source domain such that the DARC policy outputs 0 for the 0th-index. From this perspective, DARC is restricting the action support in the source domain to align the target domain action support. However, on the opposite, where the source has less action support, such methods don’t work well, and we need to account for the unseen/unsupported action in the target domain. \n\n3)  **DARAIL relaxes the assumption made in previous works that the optimal policy will receive a similar reward in both domains.** As we discussed in the general response, the assumption made by DARC guarantees that DARC’s performance and generated trajectories in the target are similar to those in the source domain. This assumption may not hold in other off-dynamics settings. We want to relax this assumption by not restricting the performance of target optimal policy in the source domain. Note that the DARC source domain trajectories resemble the optimal target ones, according to DARC’s learning objective. Thus, we use imitation learning to transfer such policy. And the experiment shows the effectiveness of our method when their assumption fails. \n\n5) **Two domains have the same reward function.** It means that $r_{\\text{src}}(s_t,a_t) = r_{\\text{trg}}(s_t,a_t)$ not $r(s)$. We will make it clearer in the revision. \n\n6) **Clarifications on Figure 1(a) and performance degradation.** To show the performance degradation empirically in more general settings, we employ DARC in the target domain and compare it with its performance in the source in Figure 1(a). Here, the motivation is, according to the DARC’s objective, that the DARC policy on source should achieve a similar performance (and a similar trajectory distribution) to the optimal policy on target. But we agree, empirically, it may not be true. So, we followed the reviewer’s suggestion to add the reward for the optimal policy on target to show the performance degradation better in Figure 1, Table 2 and 3. In contrast, little performance degradation happens in the DARC setting. We will update the figures and tables accordingly in our revision.\n\n7) **Not trained for the same steps.** We train the baselines until convergence. Though we believe the result would not change much with more training steps, we will train the baselines to the same number of steps for fair comparison in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gq756CuChu", "reviewer": "Reviewer_JwpF", "summary": "This work introduces an improvement over the previous method (DARC) with theoretical and experimental contributions for solving the off-dynamics transfer learning problem.", "questions": "Questions about the paper:\nIn the appendix you show that lower discriminator update steps are better, stopping at 50. Can you explain why lower might be better; and why not decrease even further?\n- You state a few times that you do not collect reward data from the target domain. Isn't this trivial since the reward function is identical to that of the source?\n\n\nVarious (some minor) presentation issues and references:\n- suggest to abbreviate as \"GAIL-O\" or similar. Readers will likely know GAIL and thus understand the connection more quickly\n- In main contributions, the first bullet point is quite unclear; can you re-word/shorten? Similarly, I believe you can remove all but the last sentence in bullet point 2.\n- \"works\" -> \"work\" throughout\n- Use of discounting: You use SAC with a discount factor, but $\\gamma$ is missing from the main text and appendix in many equations. Can you include / comment on this? It appears the theory is only relevant for $\\gamma=1$ which is okay but should be stated upfront.\n- Relatedly, it may be beneficial to cite in Sect. 3.1 e.g. https://arxiv.org/abs/1805.00909 ;  https://arxiv.org/pdf/2005.01643 \n- Regarding the dynamics-shift problem, can you also comment on the relationship to e.g. https://proceedings.neurips.cc/paper_files/paper/2023/file/67dd6a41bf9539cffc0fc0165e4d0616-Paper-Conference.pdf ; https://ojs.aaai.org/index.php/AAAI/article/view/25817 ; https://proceedings.mlr.press/v155/pertsch21a/pertsch21a.pdf \n- L80 \"number of data\" -> \"amount of data\"\n- L84: \"to mimic **an** expert\"\n- (nitpicking) L90: discussion of observations vs. states does not sound quite right; I believe the true distinction is in the POMDP framework\n- Eq. 3.1 remove the extra parentheses around product.\n- Eq. 3.3 Do you still need a min on the RHS? (I might be missing something, please explain if so!)\n- the value of $c$ may be nice to see explicitly so that we can see independence on the dynamics and policy.\n- L128: \"Domian\" -> \"Domain\"\n- L135 \"while we do not have much access to the target domain.\" can you be more specific?\n- L150: \"followed\" -> \"follows\"\n- L172: \"Baye's\" -> \"Bayes'\"\n- Algorithm 1; maybe highlight which parts of the algorithm are distinct from DARC?\n- Theorem 4.1: I believe it would be helpful to either (a) give a more formal version of the theorem with all definitions (define H, D_omega, is $\\gamma$ missing?) or (b) make it an \"informal\" theorem, with the full result in the appendix.\n- Which version of the environments are you using?\n- L304: \"DARAIL's ??? on Different...\" is there a missing word (Effect)?\n- Figures: Can you please include a shaded region for the DARC eval/training reward lines? \n- Figure 3: DARRIL- -> DARAIL?\n\n\n\n----\nOverall, I believe the paper needs a significant clean-up of writing and additional experiments and discussion to merit a higher score.", "limitations": "As stated above, two important limitations the authors can further comment on are (a) use of samples in the target domain and (b) gain in performance.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Using IL on states is a sensible idea to add\n- The theoretical contribution is quite strong, as the authors were able to replace a very limiting assumption with a possibly more relaxed (and simpler to analyze) assumption\n- There indeed appears to be a performance boost in the environments / transfer schemes tested.", "weaknesses": "- It's not clear how many samples are used as rollout in the target domain. If it is a non-trivial amount, then a baseline should be to run RL in the target domain with this sample budget.\n- Similarly, DARC should also be allowed to have these additional samples in the target domain for proper comparison.\n- Finally, the presentation quality poses a bit of a weakness. It is a bit difficult to read/analyze at times (see next section for suggestions to improve)\n- Figure 3 does not appear to be a significant performance boost, can you comment on the purported gain?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all of your points. \n\n---\n>How many samples are used as rollouts in the target domain?\n\nOur imitation learning step rolls out data from the target domain every 100 steps of the source domain rollouts, which is 1\\% of the source domain rollouts. \n\n---\n>Additional samples from the target domain for DARC. \n\nDARAIL does have more target domain rollouts. However, as mentioned in the general response, more rollouts of the target domain data is not the reason why DARAIL works. More rollouts of the target domain will not improve DARC performance, according to our analysis of why DARC fails in more general settings in the general response. As mentioned in the third bullet point in the general response, we compare DARC and DARAIL with the same amount of rollouts from the target domain in Tables 4 and 5 in the attached PDF, indicating that more target domain rollouts does not yield significant improvement in DARC's performance due to its inherent suboptimality. Notably, DARAIL consistently outperforms DARC even when subjected to comparable levels of target rollouts.\n\n---\n>Explanation of Figure 3.\n\nFigure 3 in the paper is an ablation study showing that our method works well under different scales of off-dynamics shift. Specifically, we conducted experiments on Ant-v2, and the target environment is broken with probability 0.2,0.5,0.8, respectively. The smaller the broken probability is, the smaller the dynamic shift is. \n\n---\n>Smaller discriminator update steps are better.\n\nIn GAIL or, more generally, GAN, the discriminator serves as the local reward function for the policy training. The smaller discriminator update steps (higher update frequency) probably lead to better discriminator training (better local reward estimation). In our experiments, we did the grid search for the parameters and noticed that further decreasing the update steps (increasing the update frequency) would not improve the performance; it is probably already well-trained, so we stopped that at 50. We will provide a more thorough grid search results to justify this in the future. \n\n---\n>Presentation comments.\n\nWe thank the reviewer for the detailed comments that help us improve the readability of the paper. We will follow your suggestions in our revision. For several questions, we briefly answer as follows:\n1) **Missing discount factor $\\gamma$ in equations.** We will state clearly in the revision that we use $\\gamma = 1$ in the analysis of the error bound. \n2) **Discuss of the related work.** [1] is most related to our problem. However, they assume that the optimal policy in different domains has similar stationary state distribution, thus, they learn a policy in the source domain and regularize the state distribution in two domains. This method is similar to DARC, which tries to learn a policy that behaves similarly in both domains. However, when the assumption is violated, the learned policy will have performance degradation when deployed to the target domain, similar to DARC. In contrast, our method doesn’t require such an assumption and we propose one more imitation learning step to transfer the policy to avoid performance degradation. [2] and [3] are designed to solve a slightly different problem that transfers prior knowledge to solve a new task instead of just considering the dynamic shift between the tasks.  We will add the discussion of these works to the related work in the revision.\n3) **Discussion of observations vs. states in L90.** The observation here represents the state trajectories we obtain from the environment, which is a general terminology from imitation learning from observation. We use imitation learning from observation to distinguish from generative adversarial imitation learning (GAIL) which uses $(s_t,a_t)$ as the expert demonstration. Meanwhile, imitation learning from observation purely learns from the observation (state observation) from the environment, i.e. $(s_t,s_{t+1})$. \n4) **Right-hand side needs a min for Eq 3.3.** The right-hand side of Eq 3.3 is the mathematical expansion of the reverse KL divergence loss. We will add a min to that to make it clearer.\n5) **Meaning of do not have much access to the target domain.** The limited access means that we cannot freely roll out infinite data from the target domain, and we can only roll out a small amount of data from the target domain. This is a standard setup shared by many off-dynamics problems, including DARC. Also, we assume we do not query the target domain reward during the training. As discussed in the first bullet point, we roll out the states and actions from the target domain data every 100 steps of the source domain rollouts. We will make it more clear in the revision.\n\nWe appreciate the reviewer's feedback. We hope we have thoroughly addressed your concerns. We are willing to answer any additional questions.\n\n---\n**References**\n\n[1] State Regularized Policy Optimization on Data with Dynamics Shift.\n\n[2] Accelerating Reinforcement Learning with Learned Skill Priors.\n\n[3] Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all reviewers for their time and constructive feedback on our paper. Since some reviewers referred to similar concerns, we would like to make a general response to address these questions.

---
> Experimental results on more general off-dynamics settings and more baselines (R_mGy8 and R_VFUe).

In the attached file, we provided additional experiment results on HalfCheetah and Reacher to validate our algorithm in general off-dynamics settings. The off-dynamics shift is constructed by modifying MuJoCo's configuration files. We modify the coefficient of gravity from 1.0 to 0.5 for the target domain. Our results in Table 1 of the attached PDF file show the effectiveness of our method. In our revision, we will add more experiments on these more general cases to demonstrate the performance of DARAIL. 

We also validate the DARAIL in intact source and broken target environments (DARC settings). The results are shown in Table 3 of the attached file. We can see that DARAIL inherits the good performance of DARC in this setting. Considering the limited settings in DARC and the more general settings in our experiment, **DARAIL will not degrade the performance of DARC in the source intact and target broken settings and will improve the performance in more general settings.** 

Further, as suggested by R_mGy8 and R_VFUe, in Table 2 and 3 of the attached file, we compare with the performance of the target optimal policy to better show the suboptimality of DARC. Also, we compare DARAIL with mimicking the $(s_t,s_{t+1})$ of the optimal trajectories in the source domain and show the superiority of DARAIL.

---
> Regarding “why DARC fails” in more general off-dynamics cases (R_LdFt and R_VFUe).

We will explain it from theoretical and empirical aspects and compare the two settings: broken source and intact target v.s. intact source and broken target.

Theoretically, as stated in Lemma B.1 in the DARC paper, the objective of DARC is equivalent to training a policy (maximizing the cumulative reward) in the source domain under a constraint that DARC behaves similarly in both domains and thus receives similar rewards. Also, in the analysis of the error bound in DARC, they made a strong assumption in Assumption 1 that **optimal policy for the target domain should also receive a similar reward in the source domain**, that is $E_{p_{\text{src}}, \pi^*}[\sum_t r(s_t,a_t)] -E_{p_{\text{trg}}, \pi^*}[\sum_t r(s_t,a_t)] \leq 4R_{max} \sqrt(\epsilon/2)$, where the $\pi^*$ is the optimal policy in the target domain, the $R_{max}$ is the maximum reward of a trajectory, $\epsilon$ is the slack term and the error bound depends on the $\epsilon$. In the setting where the source is intact and the target is broken (same as in DARC paper), the optimal policy for the target domain gives a 0 value for the 0th-index. Thus, deploying this policy in the intact source domain will receive the same reward. And the assumption here is perfectly satisfied with $\epsilon=0$. Thus, the error bound for the optimal DARC policy is 0, as analyzed in Theorem 4.1 in the DARC paper. 

Empirically, we notice that under the DARC setting, the DARC policy learns a near 0 value for the 0th-index. This guarantees that the policy can generate similar trajectories in the two domains. Also, maximizing the adjusted cumulative reward in the source domain with a policy with a near 0 value for the 0th-index is equivalent to maximizing the cumulative reward in the target domain. Thus, DARC perfectly suits the source intact and target broken setting.

However, in the source broken and target intact setting, and also other more general off-dynamics settings, the optimal policy in the target environment might not perform well in the source domain. It may even have a large performance degradation, like in the source broken and target intact settings, the policy loses one action dimension in the source domain. Thus, the assumption made in the DARC paper might not hold or, formally speaking, the $\epsilon$ there can be very large, leading to performance degradation of DARC in the target domain.

In particular, intuitively, in the source broken and target intact settings, we agree with R_VFUe that the DARC fails as it learns an arbitrary value for the 0-th dimension of action torque, which becomes detrimental in the target domain. However, this is just an artifact of the particular setting. As we discussed above, the intrinsic reason that DARC fails is the violation of the assumption. In Table 1 of the provided PDF, we demonstrate DARC’s failure in other off-dynamics settings where the dynamics shift is not induced by the broken action. 

---
> Regarding access to target domain data (R_JwpF and R_VFUe).

Both DARC and DARAIL require some limited access to the target rollouts. In DARAIL, the imitation learning step only rolls out data from the target domain every 100 steps of the source domain rollouts, which is 1\% of the source domain rollouts. We claim that more target domain rollouts will not improve DARC’s performance due to its suboptimality, and DARAIL is better not because of having more target domain rollouts. We verify it by comparing DARC and DARAIL with the same amount of rollouts from the target domain in Tables 4 and 5 in the attached file. Specifically, we examine DARAIL with 5e4 target rollouts alongside DARC with 2e4 and 5e4 target rollouts. DARAIL has 5e3 target rollouts for the Reacher environment, while DARC has 3e3 and 5e3 rollouts. From the results, we see that increasing the target rollouts from 2e4 to 5e4 (or from 3e3 to 5e3 in the case of Reacher) does not yield a significant improvement in DARC's performance due to its inherent suboptimality, as mentioned in the second bullet. Notably, DARAIL consistently outperforms DARC when given comparable levels of target rollouts.

Meta-review: This paper proposes an RL method for off-dynamics RL (adapting to new environments) by combining reward augmentation (as done in prior work) with imitation (novel contribution). The reviewers generally lean towards accepting the paper: scores = 7/6/5/5. The reviewers appreciated the importance of the problem, the clear motivation, thorough experiments, and the theoretical results. Several reviewers had writing suggestions, which the authors have promised to make. During the rebuttal period, the authors and reviewers had a thorough discussion, which seems to have resolved most of the concerns the reviewers initially had about the paper (e.g., with additional experiments). Taking this together, I recommend that the paper be accepted.
Decision: Accept (poster)