Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation

Authors: yihong  guo
yixuan  wang
yuanyuan  shi
pan  xu
anqi  liu
san  diego
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:22:24

Abstract: Training a policy in a source domain for deployment in the target domain under a dynamics shift can be challenging, often resulting in performance degradation. Previous work tackles this challenge by training on the source domain with modified rewards derived by matching distributions between the source and the target optimal trajectories. However, pure modified rewards only ensure the behavior of the learned policy in the source domain resembles trajectories produced by the target optimal policies, which does not guarantee optimal performance when the learned policy is actually deployed to the target domain. In this work, we propose to utilize imitation learning to transfer the policy learned from the reward modification to the target domain so that the new policy can generate the same trajectories in the target domain. Our approach, Domain Adaptation and Reward Augmented Imitation Learning (DARAIL), utilizes the reward modification for domain adaptation and follows the general framework of generative adversarial imitation learning from observation (GAIfO) by applying a reward augmented estimator for the policy optimization step. Theoretically, we present an error bound for our method under a mild assumption regarding the dynamics shift to justify the motivation of our method. Empirically, our method outperforms the pure modified reward method without imitation learning and also outperforms other baselines in benchmark off-dynamics environments.

Main paper: 
 introduction: 
 The objective of reinforcement learning (RL) is to learn an optimal policy that maximizes rewards through interaction and observation of environmental feedback. However, in domains such as medical treatment [1] and autonomous driving [2] , we cannot interact with the environment freely as the errors are too costly or the amount of access to the environment is limited. Instead, we might have access to a simpler or similar source domain. This requires domain adaptation in reinforcement learning. In this paper, we study a specific problem of domain adaptation in reinforcement learning (RL), where only the dynamics (transition probability) are different in two domains. This is called off-dynamics RL [3] [4] [5] . Specifically, we focus on a problem setting in which we have limited access to rollout data from the target domain, but we do not have access to the target domain reward, following the previous off-dynamics work [3] [4] [5] .Previous work on off-dynamics RL, such as Domain Adaptation with Rewards from Classifiers (DARC) [3] and [6, 5] , focuses on training the policy in the source domain with a modified reward function that compensates for the dynamics differences. The reward modification is derived so that the distribution of the learning policy's experience in the source domain matches that of the optimal trajectories in the target domain. As a result, their experience in the source domain will 38th Conference on Neural Information Processing Systems (NeurIPS 2024). networkImportance Weights < l a t e x i t s h a 1 _ b a s e 6 4 = " v f + M O W C X l X k D 7 C B / Z j 9 t q m 3 h y T 0 = " > A A A B 9 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 B I t Q U U o i U j 0 W v X i s Y D + g D W G z 3 b Z L N 5 u 4 O y m U 0 N / h x Y M i X v 0 x 3 v w 3 b t s c t P p g 4 P H e D D P z g l h w j Y 7 z Z e V W V t f W N / K b h a 3 t n d 2 9 4 v 5 B U 0 e J o q x B I x G p d k A 0 E 1 y y B n I U r B 0 r R s J A s F Y w u p 3 5 r T F T m k f y A S c x 8 0 I y k L z P K U E j e W X t 4 7 n 2 U z x z p 6 d + s e R U n D n s v 8 T N S A k y 1 P 3 i Z 7 c X 0 S R k E q k g W n d c J 0 Y v J Q o 5 F W x a 6 C a a x Y S O y I B 1 D J U k Z N p L 5 0 d P 7 R O j 9 O x + p E x J t O f q z 4 m U h F p P w s B 0 h g S H e t m b i f 9 5 n Q T 7 1 1 7 K Z Z w g k 3 S x q J 8 I G y N 7 l o D d 4 4 p R F B N D C F X c 3 G r T I V G E o s m p Y E J w l 1 / + S 5 o X F b d a q d 5 f l m o 3 W R x 5 O I J j K I M L V 1 C D O 6 h D A y g 8 w h O 8 w K s 1 t p 6 t N + t 9 0 Z q z s p l D + A X r 4 x u n f J F k < / l a t e x i t > (st, st+1) Importance Weights Figure 1: (a) Training reward in the source domain, i.e. E πDARC,p src [ t r(s t , a t )], evaluation reward in the target domain, i.e. E πDARC,p trg [ t r(s t , a t )] and optimal reward in target domain, for DARC in Ant. Evaluating the trained DARC policy in the target domain will cause performance degradation compared with its training reward, which should be close to the optimal reward in the target given DARC's objective function. Results of HalfCheetah, Walker2d, and Reacher are in Figure 9 in Appendix. (b) Learning framework of DARAIL. DARC Training: we first train the DARC in the source domain with a modified reward that is derived from the minimization of the reverse divergence between optimal policies on target and learned policies on the source. Details of DARC and the modified reward are in Section 3.1 and Appendix A.1. Discriminator training: the discriminator is trained to classify whether the data is from the expert demonstration (DARC trajectories) and provide a local reward function for policy learning. Generator training: the policy is updated with augmented reward estimation, which integrates the reward from the source domain and information from the discriminator. We first train DARC, collect DARC trajectories from the source domain, and then train the discriminator and the generator alternatively. produce a trajectory distribution close to the target domain's optimal one. However, deploying the resulting policy in the target domain usually causes performance degradation compared to its training performance in the source domain. Figure 1 (a) shows the experiment result of DARC under a broken source environment setting, where the broken source environment means the value of 0-index in the action of the source domain is frozen to 0, and the target environment remains intact. Consequently, existing reward modification methods will only obtain a sub-optimal policy in the target domain. Details of DARC and its suboptimality in the target domain will be introduced in Section 3. 1 . More details about why DARC fails in more general dynamics shift cases are in Appendix C. 6 .In this paper, we present an off-dynamics reinforcement learning algorithm described in Figure 1 (b) . Our method, Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) consists of two components. Following previous work like DARC [3] on off-dynamics RL, we first obtain the source domain trajectories that resemble the target domain's optimal ones. We then transfer the policy's behavior from the source to the target domain through imitation learning from observation [7] , which can mimic the policy's behavior from the state space.In particular, we consider the dynamics shift in the framework of generative adversarial imitation from observation (GAIfo) [8] , and propose a novel and practical reward estimator called the reward augmented estimator (R AE ) for the policy optimization step in imitation learning.Our contributions can be summarized as follows:• We propose the Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) algorithm by transferring the learned policy of reward modification approaches from the source domain to the target domain via mimicking state-space trajectories in the source domain. We propose reward augmented estimator (R AE ) to leverage the reward from the source domain to stabilize the learning. • We recognize limitations in the existing DARC algorithm and off-dynamics reinforcement learning algorithms with similar reward modification, which is directly deploying the learned policy to the target domain results in significant performance degradation. Our proposed algorithm mitigates this issue with an imitation learning component that transfers DARC policy to the target. • We introduce an error bound for DARAIL that relaxes the assumption made in previous works that the optimal policy will receive a similar reward in both domains. Specifically, with our imitation learning from the observation component, we can show the convergence of DARAIL with a mild assumption on the magnitude of the dynamics shift. • We conducted experiments on four Mujoco environments, namely, HalfCheetah, Ant, Walker2d, andReacher on modified gravity/density configurations and broken action environments. A comparative analysis between DARAIL and baseline methods is performed, demonstrating the effectiveness of our approach. Our method exhibits superior performance compared to the pure modified reward method without imitation learning and outperforms other baselines in these environments. Code is available at https://github.com/guoyihonggyh/Off-Dynamics-Reinforcement-Learning-via-Domain-Adaptation-and-Reward-Augmented-Imitation. 
backgrounds: 
 Off-dynamics reinforcement learning We consider two Markov Decision Processes (MDPs): one is the source domain M src , defined by (S, A, R, p src , γ), and the other one is the target domain M trg , defined by (S, A, R, p trg , γ). The difference between them is the dynamics p, also known as transition probability, i.e., p src ̸ = p trg or p src (s t+1 |s t , a t ) ̸ = p trg (s t+1 |s t , a t ). In our paper, we experiment with two types of dynamics shift: 1) broken environment [3] , in which the 0-th index value is set to be 0 in action, and 2) modifying the gravity/density setting of the target environment [9] . The source and the target domain share the same reward function, i.e., r src (s t , s t+1 ) = r trg (s t , s t+1 ). All other settings, including state space S, action space A, and the discounting factor γ, are the same. We will use γ = 1 in the derivation and analysis in our paper.We aim to learn a policy ζ(a|s) using interaction from the source domain together with a small amount of data from the target domain (s t , a t , s t+1 ) trg to maximize the expected discounted sum of reward E ζ,ptrg [ t γ t r(s t , a t )] in the target domain. Note that we assume we only have limited access to the target domain transition, namely (s t , a t , s t+1 ) trg , in the whole process and we do not utilize the target domain reward.Imitation learning (from Observation) Imitation Learning (IL) trains a policy to mimic an expert policy π E with expert demonstration {(s 0 , a 0 ), (s 1 , a 1 ), ...} or {(s 0 , s 1 ), (s 1 , s 2 ), ...}. Generative adversarial imitation learning (GAIL) [7] uses an objective similar to Generative adversarial networks (GANs) that minimizes the distribution generated by the policy and the expert demonstration. It alternatively trains a discriminator D ω and a policy π θ to solve the min-max problem:min π θ max Dω E (s,s ′ )∼π E log D ω (s, s ′ ) + E (s,s ′ )∼π θ log(1 -D ω (s, s ′ )) -λH(π θ ), (2.1)where s ′ is the next state and H(π θ ) is the entropy of the policy π θ . Note that in our problem, we mimic the state-only expert demonstrations {(s 0 , s 1 ), (s 1 , s 2 ), ...} instead of the expert's actions. This setting is also called imitation learning from observation [8] . We will further discuss why we use state observation instead of action in section 3.2. D ω is the classifier that discriminates whether the state pair is from the expert π E or generated by the policy π θ . Then, the policy is trained with the RL algorithm using reward estimation -log D ω (s, s ′ ) as the reward. The optimization of the Eq. (2.1) involves alternatively training the policy and the discriminator. 
off-dynamics rl via domain adaptation and reward augmented imitation learning: 
 In this section, we present our algorithm, DARAIL, under the off-dynamics RL problem setting. First, we introduce DARC [3] in Section 3.1, which provides the distribution of target optimal trajectories in the source domain to mimic. Then, in Section 3.2, we introduce the imitation learning component through which we utilize the trajectories provided by DARC and transfer the DARC policy to the target domain. We aim to learn a policy that generates the same distribution of trajectories in the target domain as the DARC trajectories in the source domain.DARC is proposed to solve the off-dynamics RL through a modified reward that compensates for the dynamics shift [3] . Here, we first introduce DARC and its drawbacks. DARC seeks to match the policy's experiences in the source domain and optimal trajectories in the target domain. We define τ = {(s 1 , a 1 ), (s 2 , a 2 ), ..., (s t , a t ), ...} as a trajectory. We use τ src π θ to represent the trajectories generated by π θ in the source domain. The policy's distribution over trajectories in the source domain is defined as:EQUATIONLet π * = argmax π E π,ptrg [ t r(s t , a t )] be the policy maximizing the cumulative reward in the target domain. We use τ trg π * to represent the trajectories generated by π * in the target domain. Given the assumption that the optimal policy π * in the target domain is proportional to the exponential reward, i.e., π * (a t |s t ) ∝ exp( t r(s t , a t )), the desired distribution over trajectories in the target domain is defined as:EQUATIONDARC policy can be obtained by minimizing the reverse KL divergence of p(τ trg π * ) and q(τ src π θ ):EQUATIONwhere ∆r(s t , a t , s t+1 ) := log p trg (s t+1 |s t , a t ) -log p src (s t+1 |s t , a t ) and c is a partition function of p(τ trg π * ), which is independent of the dynamics and policy. The ∆r(s t , a t , s t+1 ) can be calculated through the following procedure: i), train two classifiers p(trg|s t , a t ) and p(trg|s t , a t , s t+1 ) with cross-entropy loss L CE ; ii), Use Bayes' rules to obtain the log ptrg(st+1|st,at) psrc(st+1|st,at) . Details are in Appendix C.1. Eq. (3.3) shows that π DARC can be obtained via maximum entropy algorithm with a modified reward r modified = r(s t , a t ) + ∆r(s t , a t , s t+1 ) at every step.However, DARC matches the distribution of τ trg π * and τ src πDARC . As the dynamics shift exists, π DARC will not recover the optimal policy π * , and deploying the DARC in the target domain will usually suffer from performance degradation due to the dynamics shift, as shown in Figure 1 (a) and Figure 9 in Appendix. However, in the source domain τ src πDARC resembles those optimal trajectories in the target domain. Given the property of τ src πDARC , we propose to use imitation learning from observation with τ src πDARC as expert demonstrations to transfer DARC to the target domain. The new policy in the target domain should behave similarly (generate similar trajectories) as DARC in the source domain.In this section, we present the Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) method, which mitigates the problem of DARC via imitation learning from observation. As described in Section 3.1, τ src πDARC resembles the target optimal trajectories, and we want to transfer DARC's behavior to the target domain. A natural way to tackle it is utilizing imitation learning to mimic the expert demonstration τ src πDARC . Following [7, 8] , the objective can be formulated as:min ζ maxD ω E ptrg,ζ t log Dω(st, st+1) + E (s t ,s t+1 )∼τ src π DARC t log(1 -Dω(st, st+1)) . (3.4)where D ω is the discriminator in the generative adversarial imitation learning and ζ is the policy to be learned in the target domain. In the objective function Eq. (3.4), the (s t , s t+1 ) pairs are from the target domain, while we do not have much access to the target domain. Alternatively, we can use the (s t , s t+1 ) pairs from the source domain and re-weight the transition with the importance sampling method to account for the dynamics shift. The objective with data rolled out from the source domain, and the importance sampling is as follows:EQUATIONwhere ρ(s t , s t+1 ) = ptrg(st+1|st,at) psrc(st+1|st,at) is the importance weight. Note that we do the generative adversarial imitation learning from only state observations (GAILfo) with (s t , s t+1 ) [9] [10] [11] instead of (s t , a t ). This is because we aim to learn a policy ζ to produce the same trajectory distributions in the target as the ones π DARC produces in the source domain, despite the dynamics shift, rather than mimicking the policy. Mimicking the (s t , a t ) pairs will recover the same policy as DARC, and deploying it to the target domain will not recover the expert trajectories due to the dynamics shift. This objective Eq. (3.5) can be interpreted as training the discriminator D ω to discriminate whether the (s t , s t+1 ) generated by ζ in the target domain matches the distribution of DARC trajectories in the source domain using data rolled out from the source domain with ζ and importance weight. Then, after the discriminator is fitted, the policy can be trained with the reward estimator R AE with model-free RL. The objective is:EQUATIONwhere R AE is defined as follows: R AE (s t , s t+1 ) = -log D ω (s t , s t+1 ) + ρ(s t , s t+1 )(r src (s t , s t+1 ) + log D ω (s t , s t+1 )).(3.7)Here the r src (s t , s t+1 ) is the reward obtained from the source domain, which is the same as the reward from the source domain, i.e. r trg (s t , s t+1 ). In imitation learning, the -log D ω (s t , s t+1 ) can be viewed as a local reward function for the policy optimization step and the objective ismax ζ E psrc,ζ [ t -log D ω (s t , s t+1 )]. So Eq.(3.5) can be viewed as learning a reward function for the training of ζ. However, as the dynamics shift exists, the estimation of the -log D ω (s t , s t+1 ) could be biased, which is similar to the case in off-policy evaluation (OPE) [12] [13] [14] [15] [16] when training a reward estimation on biased data. As we have access to the source domain and can obtain the reward from the rollout, we are motivated to use both the reward estimation -log D ω (s t , s t+1 ) and the ground truth reward in the source domain r src (s t , s t+1 ) so that we could have a better reward estimation than -log D ω (s t , s t+1 ) under dynamics shift. The R AE here can be viewed as using log D ω (s t , s t+1 ) as a base estimator of the reward and use r src (s t , s t+1 ) and importance weight ρ(s t , s t+1 ) to correct it. This correction idea is similar to the doubly robust estimator (DR) [12] in OPE. The DR estimator combines the reward estimation r and the importance-weighted difference between true reward r and r. Specifically, the DR method takes the reward estimation r as a base estimator and applies the importance weighting to the difference between true reward r and r, which is ρ(rr) term, to correct the bias of the r, where ρ is the importance weight. Our Algorithm The DARAIL is shown in Algorithm 1, which consists of two steps: the first step, Line 2 in Algorithm 1, is the training of π DARC , and the second step is imitation learning with the reward estimator in Eq. (3.7). In Lines 6-8, we roll out the target domain transition (s t , a t , s t+1 ) to calculate the importance weight. Here, we will not collect the target domain reward. In Lines 9-11, we update the discriminator based on Eq. (3.5). In Line 12, we train the two classifiers p(trg|s t , a t ) and p(trg|s t , a t , s t+1 ) with cross-entropy loss L CE and Bayes' rules similar to ∆r(s t , a t , s t+1 ) in DARC as mentioned in Section 3.1. The details are in Appendix C.1. Lastly, we calculate the R AE in Line 13 and update the generator (Soft Actor-Critic (SAC) [17] ) with R AE in Line 14.Note that in Lines 6-7, we roll out from the target domain, but the amount of it is significantly smaller than the source rollouts. In our experiments, we roll out from the target domain every 100 steps of source domain rollouts, which is 1% of the source domain rollouts. Further, even though DARAIL requires more target domain rollouts than DARC as it is required to train DARC first and then perform the imitation learning step, the advantage of DARAIL does not solely come from the more target samples. Because, in DARC, increasing the training step or target domain rollouts will not further improve its performance due to its inherent suboptimality, which is shown in table 11 and 12 in Appendix with the same amount of target domain rollouts. 
theoretical analysis of darail: 
 Let π * = argmax π E π,ptrg [ t r(s t , a t )] be the optimal policy maximizing the cumulative reward in the target domain and ζ be the policy learned from DARAIL. Now, we provide an error bound for DARAIL. Details of the proof are deferred to Appendix B. ) -inf ζ d D (τ src πDARC , τ trg ζ ) ≤ ε, ∀ δ ∈ (0, 1), with probability at least 1 -δ, we have 2) estimation error. This bound demonstrates how the two important components in our proposed approach contribute to a good performance. Firstly, we would want a well-trained policy on the source to reduce (1), which can be achieved by a good policy learning algorithm and well-trained classifiers for reward modification. Secondly, we utilize imitation learning from observation to transfer the experience to the source. (2.1) depends on the upper bound of the importance weight, which can be decreased with a richer policy class or when the dynamics shift becomes smaller. Additionally, a better imitation can be also achieved by increasing the complexity of the discriminator function class and the number of samples, which pushes (2.2) to be smaller.EQUATIONAs we discussed in Section 3.1, the DARC algorithm [3] trains a policy π DARC on the source domain via matching the distribution of trajectories generated by π DARC in the source and the distribution of the optimal trajectory in the target domain. Consequently, the learned policy π DARC will be suboptimal if it is directly deployed in the target domain.In the DARC analysis, it is assumed that the optimal policy for the target domain π * lies in the no exploit set defined as follows [3, Assumption 1] .Π no exploit ≜ E a∼π(a|s) t D KL (p src (s t+1 |s t , a t )||p trg (s t+1 |s t , a t )) ≤ ϵ . (4.1)Here, the no exploit set means that the experiences for any policy in this set are similar in the source and target domains. Consequently, any two policies in this no exploit set also receive similar expected rewards in the two domains, and thus the reward received by π * in the target domain is similar to that received by π DARC in the target domain. Further, the objective function Eq. (3.3) of DARC is equivalent to the following constrained optimization.max π∈Πno exploit E psrc,π t r(s t , a t ) + H[a t |s t ] . (4.2)Thus, deploying the policy π DARC will not receive a huge performance degradation. However, the assumption that π * ∈ Π no exploit is stringent and might not always be satisfied when the dynamics shift is large. When this assumption is violated, π * is not a good policy in the source domain, though it is the optimal policy in the target domain. Thus, the DARC policy which only optimizes the modified reward in the source domain will have significant performance degradation, as we have empirically shown in Figure 1 (a) and Figure 9 . We also demonstrate this performance gap in Lemma A.1 in Appendix A when their assumption is not satisfied.In contrast, our algorithm DARAIL does not assume the performance of π DARC in the source domain to be close to the performance of π * in the target domain. Instead, we only assume that the importance weight is somehow bounded, meaning that the dynamics shift is bounded. The error bound of our algorithm presented in Theorem 4.1 is controlled by imitation learning, which transfers the performance of π DARC in the source domain to that of π * in the target domain without assuming π * ∈ Π no exploit . Therefore, our algorithm can work well even in the cases shown in Figure 1 (a) and Figure 9 where the experience of π DARC is very distinctive in the source and target domains. 
experiment: 
 In this section, we conduct experiments on off-dynamics reinforcement learning settings on four OpenAI environments: HalfCheetah-v2, Ant-v2, Walker2d-v2, and Reacher-v2. We compare our method with seven baselines and demonstrate the superiority of the proposed DARAIL.Dynamics Shifts: We examine our algorithm with two types of dynamics shift. 1) Broken environment. Following previous work [3] , we freeze the 0-index value to 0 in action: zero torque is applied to this joint, regardless of the commanded torque. Different from DARC [3] , who only test their method in intact source and broken target environment, we further test our algorithm in the broken source and intact target environment, where the source has less support than the target domain. As discussed in Section 4.1, violating the π * ∈ Π no exploit assumption leads to significant performance degradation for DARC and similar methods. When the source domain is intact, this assumption is more likely to hold and DARC can achieve a near-optimal policy in the target domain. So, besides the setting in DARC, we focus on a harder problem for off-dynamics RL where DARC is prone to failure due to the violation of the assumptions in Section 4.1. Further, for the Ant and Walker2d, the source environment is broken with p f = 0.8 probability, which means that with 0.8 probability, the 0-index will be set to be 0, and 0.2 probability remains the original value. More details about the broken environment will be introduced in the Appendix C.3. 2) Modify parameters of the environment.Besides the broken environment, we create dynamics shifts by modifying MuJoCo's configuration files for the target domain. Specifically, we modify one of the coefficients of {gravity, density} from 1.0 to one of the value {0.5, 1.5}.Baselines: We first compare our method with DARC performance in the source and target domains. DARC Training and DARC Evaluation, defined as E psrc,πDARC [ t r(s t , a t )] and E ptrg,πDARC [ t r(s t , a t )] respectively, represent DARC performance in the two domains. We compare DARAIL with DARC training performance as we mimic the DARC behavior in the source domain, which should receive a similar reward as the DARC training reward in the source domain. We compare with DARC Evaluation to show that our method mitigates the problem of DARC and outperforms DARC in the target domain. Further, we compare our method DARAIL with several baselines that we describe as follows. Importance Sampling for Reward (IS-R) re-weights the reward in the transition with ptrg(st+1|st,at) psrc(st+1|st,at) , and update the policy with reward ptrg(st+1|st,at) psrc(st+1|st,at) r(s t , a t ) [18] . Importance Sampling for SAC Actor and Critic Loss (IS-ACL) [18] re-weights the transitions in the SAC actor and critic loss. DAIL is a reduction of DARAIL without reward augmentation. Model-based RL method MBPO [19] uses short model rollouts branched from real data to reduce the compounding errors of inaccurate models and decouple the model horizon from the task horizon. MATL [20] uses different modified rewards and is similar to our problem setting, except that they have access to rewards in the target domain. Finally, we compare with generative adversarial reinforced action transformation (GARAT) [10] , a grounded action transformation method that uses imitation learning to modify the action that is executed in the source domain to simulate the target transitions. More details of the baselines are in Appendix C.2.Experimental Details: We perform weight clipping to all methods that use the importance weight, including the DARAIL, DAIL, IS-R, and IS-ACL, and select the [0.01, 100] as the clipping interval for fair comparison, which works well for all methods. We also show that DARAIL is less sensitive to the importance of weight clipping in the next section. We conduct fair parameter tuning for our method and baselines, including learning rate, Gaussian noise scale, and learning frequency of the importance weight. We also tune the parameter for the imitation learning component in DARAIL and DAIL and notice that the higher update frequency tends to perform better, and experiment results are in Appendix D.2. More details are in Appendix D.4.We show the results of DARAIL and DARC in Table 1 and 2 for broken source and 1.5 gravity setting, respectively. And the results of other baselines are in Table 3 and 4 . We refer to the results on other settings in the Appendix, including the intact source and broken target environment setting and the modification of different scales of the parameters in the configuration file. We will also empirically discuss why DARC works well in the broken target setting while fails in the broken source setting in Appendix C.6. the clipping interval of importance weight. In Figure 2 , we show the performance of DARAIL and IPS-ACL on different importance weight clipping intervals in the broken source setting, and DARAIL outperforms IPS-ACL on all tasks. If the clipping interval is too large, IPS-ACL suffers from high variance, thus harming the performance. If the clipping interval is too small, the effective information about the dynamics shift is lost. On the other hand, DARAIL is less sensitive to it, which is an inherent property of our R AE . Furthermore, in Figure 2 , for IPS-ACL, the training curve for [0.001, 1000] clipping interval has a much larger variance than [0.1, 10] clipping interval, while our method does not suffer from such a high variance. This also demonstrates that our proposed reward estimator R AE is a more robust estimator and less affected by the importance weight. 3 , IPS-ACL receives comparable performance with DARAIL with the clipping interval [0.01,100], while the performance decreases significantly with different intervals.In our broken action environments, as we create the off-dynamics shift by (probabilistically) freezing one action dimension in the source domain, we can control the off-dynamics shift magnitudes by controlling the broken probability.For the same environment, the larger the p f is, the higher the probability of freezing the 0-index action, thus a larger dynamics shift. We consider p f = [0.2, 0.5, 0.8] for Ant, respectively and the experiment results is shown in Figure 3 . From left to right, as the dynamics shift increases, we observe that the DARC performance decreases, and DARAIL outperforms DARC on all tasks. 
related work: 
 Off-dynamics RL Off-dynamics RL [3] is a specific domain adaptation [21, 22] and transfer learning problem in the RL domain [23] where the goal is to learn a policy from a source domain to adapt to a target domain where the dynamics are different. Similar to many works in off-policy evaluation (OPE) [12] in bandit and offline/off-policy RL [13, 24] , an importance weight approach can be used to account for the difference between the transition dynamics with ptrg(st+1|st,at) psrc(st+1|st,at) . However, this method can easily suffer from high variance due to the estimation bias of p src (s t+1 |s t , a t ) [12] . Another line of method for the off-dynamics RL is through reward shaping [3, 5] . DARC [3] learns a policy from a modified reward function that accounts for the dynamics shifts through a trajectories distribution matching objective. [6] proposed an unsupervised domain adaptation method with KL regularized objective, which uses the same reward modification techniques trajectories distribution matching objective in DARC [3] . These reward-shaping methods all face the same problem: they will not recover the optimal policy in the target domain and will suffer from performance degradation in the target domain, but the policy's experience in the source domain is similar to the optimal policy in the target domain. Similarly, [25] proposes a state-regularized policy optimization method that constrains the state distribution to be similar in the source and target domain by adding a constraint term in the reward. However, this will also lead to suboptimal policy in the target domain like DARC. Different from DARC, Mutual Alignment Transfer Learning (MATL) [20] uses different modified rewards with GAN [26] to align the trajectories generated in the source and the target domain, but it requires access to the target domain reward. There is also work [27] that solves the off-dynamics RL problem by training a distributionally robust policy in the source domain by assuming that the target domain's transition probability is in an ambiguity set defined around the transition probability of the source domain. Our method builds on DARC, inspired by its property in the source domain, overcoming the issues in DARC and similar methods by mimicking the π DARC behavior in the source domain.Imitation Learning Imitation learning (IL) is another line of work that can be applied to off-dynamics problems by mimicking the expert demonstration in the target domain. Generative adversarial imitation learning, [7, 28-30, 8, 31, 32] , frames IL as an occupancy-measure matching or divergence minimization problem, which minimizes the divergence of the generated trajectories and the expert demonstration. Building on GAN [26] , it uses the RL algorithm as a generator and a classifier as a discriminator to achieve this. Imitation learning from observation (Ifo) [33] [34] [35] is recently proposed to mimic the expert's behavior without knowing which actions the expert took. In the off-dynamics RL setting, recent work on IL under dynamics mismatch [11, 10, 36] can transfer a policy learned in the source to the target domain with minimal interaction with the target domain. However, these methods require high-quality and sufficient expert demonstrations and also the expert demonstrations might not be the optimal trajectories for the target domain, resulting in a suboptimal policy. Our method, DARAIL, transfers the DARC policy's behavior in the source to the target domain through imitation learning from observation so that the new policy will behave like the optimal policy in the target domain. Furthermore, we propose a novel and practical reward estimator with the signal from the discriminator and the reward from the source domain for the policy optimization. 
conclusion: 
 In this paper, we propose Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) for off-dynamics RL. We recognize the drawbacks of DARC and its following work with the same modified rewards function. We demonstrate that DARC or similar reward modification methods can only obtain a near-optimal policy in the target domain. We then propose to mimic the trajectory distribution generated by DARC in the source domain. Specifically, we propose a reward-augmented estimator for the policy optimization step in imitation learning from observation. Theoretically, we established the finite sample upper bounds of rewards for the proposed method, relaxing the restrictive assumption about the optimal policy in the previous work. Empirically, we conducted experiments on four Mujoco environments, demonstrating the superiority of our method. From the safety perspective, our method avoids directly training a policy in a high-risk environment. Our future work includes investigating off-dynamics reinforcement learning under safety constraints and more severe domain gaps in reinforcement learning. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We run the experiment on the simulated RL benchmarks; thus, no such issue exists.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We provide citations to all the data and related work in our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We include the code in our paper. Also, details about the implementation are included in the paper.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our experiments are conducted on the RL benchmarks and thus do not involve any crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: Our research and experiment don't require IRB as we conducted experiments on simulated RL benchmarks.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 DARC ObjectiveFigure 4 shows the objective of DARC, which minimizes the reverse KL divergence of the trajectories generated by the π DARC in the source domain and π * in the target domain. Note that the optimal policy is assumed to be proportional to the exponential form of the reward, i.e. π * ∝ exp(r(s t , a t )).Given this assumption, the reverse KL divergence can be re-formulated to Eq. (3.3) with modified reward. So, the π DARC will not be optimal in the target domain but can generate trajectories in the source domain that resemble the optimal trajectories given the objective.Minimize Reverse KL Divergence Source Domain < l a t e x i t s h a 1 _ b a s e 6 4 = " v f + M O W C X l X k D 7 C B / Z j 9 t q m 3 h y T 0 = " > A A A B 9 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 B I t Q U U o i U j 0 W v X i s Y D + g D W G z 3 b Z L N 5 u 4 O y m U 0 N / h x Y M i X v 0 x 3 vi Z 7 c X 0 S R k E q k g W n d c J 0 Y v J Q o 5 F W x a 6 C a a x Y S O y I B 1 D J U k Z N p L 5 0 d P 7 R O j 9 O x + p E x J t O f q z 4 m U h F p P w s B 0 h g S H e t m b i f 9 5 n Q T 7 1 1 7 K Z Z w g k 3 S x q J 8 I G y N 7 l o D d 4 4 p R F B N D C F X c 3 G r T I V G E o s m p Y E J w l 1 / + S 5 o X F b d a q d 5 f l m o 3 W R x 5 O I J j K I M L V 1 C D O 6 h D A y g 8 w h O 8 w K s 1 t p 6 t N + t 9 0 Z q z s p l D + A X r 4 x u n f J F k < / l a t e x i t > (s t , s t+1 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " v f + M O W C X l X k D 7 C B / Z j 9 t q m 3 h y T 0 = " > A A A B 9 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 B I t Q U U o i U j 0 W v X i s Y D + g D W G z 3 b Z L N 5 u 4 O y m U 0 N / h x Y M i X v 0 x 3 vi Z 7 c X 0 S R k E q k g W n d c J 0 Y v J Q o 5 F W x a 6 C a a x Y S O y I B 1 D J U k Z N p L 5 0 d P 7 R O j 9 O x + p E x J t O f q z 4 m U h F p P w s B 0 h g S H e t m b i f 9 5 n Q T 7 1 1 7 K Z Z w g k 3 S x q J 8 I G y N 7 l o D d 4 4 p R F B N D C F X c 3 G r T I V G E o s m p Y E J w l 1 / + S 5 o X F b d a q d 5 f l m o 3 W R x 5 O I J j K I M L V 1 C D O 6 h D A y g 8 w h O 8 w K s 1 t p 6 t N + t 9 0 Z q z s p l D + A X r 4 x u n f J F k < / l a t e x i t > (s t , s t+1 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " U m G h b 6 A X + Y w C e H 2 e 8 M n 5 l d C C l X o = " > A A A B 8 X i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 k V q R 6 r 9 e C x i v 3 A d i n Z N N u G J t k l y Q p l 6 b / w 4 k E R r / 4 b b / 4 b 0 3 Y P 2 v p g 4 P H e D D P z g p g z b V z 3 2 8 m t r K 6 t b + Q 3 C 1 v b O 7 t 7 x f 2 D p o 4 S R W i D R D x S 7 Q B r y p m k D c M M p + 1 Y U S w C T l v B q D b 1 W 0 9 U a R b J B z O O q S / w Q L K Q E W y s 9 N i N W S + 9 u b q v T X r F k l t 2 Z 0 D L x M t I C T L U e 8 W v b j 8 i i a D S E I 6 1 7 n h u b P w U K 8 M I p 5 N C N 9 E 0 x m S E B 7 R j q c S C a j + d X T x B J 1 b p o z B S t q R B M / X 3 R I q F 1 m M R 2 E 6 B z V A v e l P x P 6 + T m P D S T 5 m M E 0 M l m S 8 K E 4 5 M h K b v o z 5 T l B g + t g Q T x e y t i A y x w s T Y k A o 2 B G / x 5 W X S P C t 7 l X L l 7 r x U v c 7 i y M M R H M M p e H A B V b i F O j S A g I R n e I U 3 R z s v z r v z M W / N O d n M I f y B 8 / k D G S 6 Q j w = = < / l a t e x i t > ⇡ DARC < l a t e x i t s h a 1 _ b a s e 6 4 = " s Y K w H H g q B b 9 I c 0 6 q 1 4 f r + S R O v A s = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S K I h 5 K I V I 9 F L x 4 r m L b Q x r L Z T t u l m 0 3 Y 3 Q g l 9 D d 4 8 a C I V 3 + Q N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Y S K 4 N q 7 7 7 a y s r q 1 v b B a 2 i t s 7 u 3 v 7 p Y P D h o 5 T x d B n s Y h V K 6 Q a B Z f o G 2 4 E t h K F N A o F N s P R 7 d R v P q H S P J Y P Z p x g E N G B 5 H 3 O q L G S 3 0 n 4 4 3 m 3 V H Y r 7 g x k m X g 5 K U O O e r f 0 1 e n F L I 1 Q G i a o 1 m 3 P T U y Q U W U 4 E z g p d l K N C W U j O s C 2 p Z J G q I N s d u y E n F q l R / q x s i U N m a m / J z I a a T 2 O Q t s Z U T P U i 9 5 U / M 9 r p 6 Z / H W R c J q l B y e a L + q k g J i b T z 0 m P K 2 R G j C 2 h T H F 7 K 2 F D q i g z N p + i D c F b f H m Z N C 4 q X r V S v b 8 s 1 2 7 y O A p w D C d w B h 5 c Q Q 3 u o A 4 + M O D w D K / w 5 k j n x X l 3 P u a t K 0 4 + c w R / 4 H z + A G 4 Y j n Q = < / l a t e x i t > ⇡ ⇤Now, we show that without the assumption of π * ∈ Π no exploit in [3] , the error of π DARC cannot be trivially bounded.Lemma A. where R max refers to the maximum entropy-regularized return of any trajectories. However, the inequality Eq. (A.1) only holds for π DARC , not for π * . Now, we show that without the assumption π * ∈ Π no exploit , the error could not be bounded trivially.We start with the same decomposition. Therefore, we have In the original proof of [3] , they bound the three terms based on the following idea:E ptrg,For the term I 1 , they directly assume π * ∈ Π no exploit and obtain I 1 ≤ 2R max ϵ/2 based on inequality Eq. (A.1). However, without the π * ∈ Π no exploit , the upper bound is not valid. A valid upper bound should be: And the total variation of the two policies cannot be trivially bound as well. For the term I 3 , we can easily bound it by applying the inequality Eq. (A.1) as π DARC ∈ Π no exploit .I 1 = E ptrg,π * t r(In summary, the bound without assuming π * ∈ Π no exploit will be: E ptrg,R(n) µ = E σ [sup f ∈F ] 1 n n i=1 σ i f (x i ). (B.(D) = {c 0 + k i c i D i : c 0 ∈ R, D i ∈ D, n ∈ N}.Assuming the ground truth reward function r lies in the span(D), then the compatible coefficient is defined as:∥r∥ D = inf k i |c i | : r = c 0 + k i c i D i , c 0 ∈ R, D i ∈ D, n ∈ N . (B.2)The compatible coefficient represents the minimum number of functions in D required to the reward function r, which means the complexity of the reward function r. ) ≤ ε (empirical neural network distance achieved by imitation learning), the importance weight ρ(s, s t+1 ) is bounded by W , m is the number of the expert data, then ∀ δ ∈ (0, 1), with probability at least 1 -δ, we haveE psrc,πDARC t r(s t , a t ) -E ptrg, ζ t r(s t , a t ) ≤ ∥r D ∥ inf ζ d D (τ src πDARC , τ trg ζ ) + 2 R(m) τ src π DARC + 2W R(m) τ trg ζ + (6W + 1)∆ log(4/δ) 2m + ε . Proof. Given d D (τ src πDARC , τ trg ζ ) -inf ζ d D (τ π src DARC , τ trg ζ ) ≤ ε, we can have d D (τ src πDARC , τ trg ζ ) ≤ d D (τ src πDARC , τ trg ζ ) -d D (τ src πDARC , τ trg ζ ) + inf ζ d D (τ src πDARC , τ trg ζ ) + ε.We prove that d D (τ src πDARC , τ trg ζ ) -d D (τ src πDARC , τ trg ζ ) has an upper bound. d D (τ src πDARC , τ trg ζ ) -d D (τ src πDARC , τ trg ζ ) = sup D∈D E (st,st+1)∼τ src π DARC [D(s t , s t+1 )] -E (st,st+1)∼τ trg ζ [D(s t , s t+1 )] -sup D∈D E (st,st+1)∼τ src π DARC [D(s t , s t+1 )] -E (st,st+1)∼τ trg ζ [D(s t , s t+1 )] ≤ sup D∈D E (st,st+1)∼τ src π DARC [D(s t , s t+1 )] -E (st,st+1)∼τ src π DARC [D(s t , s t+1 )] + sup D∈D E (st,st+1)∼τ trg ζ [D(s t , s t+1 )] -E (st,st+1)∼τ trg ζ [D(s t , s t+1 )] = sup D∈D E (st,st+1)∼τ src π DARC [D(s t , s t+1 )] -E (st,≤ 2E σ,τ src π DARC sup D∈D m i=1 1 m σ i D(s i , s ′ i ) + 2∆ log(4/δ) 2m ≤ 2R (m) τ src π DARC + 2∆ log(4/δ) 2m ≤ 2 R(m) τ src π DARC + 6∆ log(4/δ) 2m .By a similar derivation, we can haveW sup D∈D E (st,st+1)∼τ src ζ [D(s t , s t+1 )] -E (st,st+1)∼τ src ζ [D(s t , s t+1 )] ≤ 2W R(m) τ src ζ + 6W ∆ log(4/δ) 2m .Thus, we haved D (τ src πDARC , τ trg ζ ) -d D (τ src πDARC , τ trg ζ ) ≤ 2 R(m) τ src π DARC + 2W R(m) τ trg ζ + (6W + 1)∆ log(4/δ) 2m .Then, based on Theorem 2 in [38] , we can conclude thatE psrc,πDARC t r(s t , a t ) -E ptrg, ζ t r(s t , a t ) ≤ ∥r D ∥ inf ζ d D (τ src πDARC , τ trg ζ ) + 2 R(m) τ src π DARC + 2W R(m) τ src ζ + (6W + 1)∆ log(4/δ) 2m + ε .This completes the proof.Theorem B.5. Let π * = argmax π E π,ptrg [ t r(s t , a t )] be the policy maximizing the cumulative reward in the target domain and ζ be the policy learned from DARAIL. Let m be the number of the expert demonstration and R(m ) π = E σ sup D∈D 1 m m i=1 σ i D(+ ∥r∥ D ε + inf ζ d D (τ src πDARC , τ trg ζ ) APPROXIMATION ERROR + 2 R(m) τ trg π DARC + 2W R(m) τ trg ζ + (6W + 1)∆ log(4/δ) 2m ESTIMATION ERROR IMITATION LEARNING ERROR BOUND .Proof. We can first decompose it into three terms: .E ptrg,π * t r(s t , a t ) -E ptrg, ζ t r(s t , a t ) = E ptrg,π * t r(s t , a t ) -E psrc,πBased on the formulation, π * DARC can generate optimal trajectories for the target domain in the source domain so that I 1 = 0. Also, the I 2 term is the training error of the DARC and the entropy term of the optimal DARC policy, and we can assume together they are bounded by B. Then, we only need to bound the I 3 terms. Combining Lemma B.4, we haveE ptrg,π * t r(s t , a t ) -E ptrg, ζ t r(s t , a t ) ≤ E psrc,π * DARC t r(s t , a t ) + H[a t |s t ] -E psrc,πDARC t r(s t , a t ) DARC ERROR BOUND IN SOURCE + ∥r∥ D ε + inf ζ d D (τ src πDARC , τ trg ζ ) APPROXIMATION ERROR + 2 R(m) τ trg π DARC + 2W R(m) τ trg ζ + (6W + 1)∆ log(4/δ) 2m ESTIMATION ERROR IMITATION LEARNING ERROR BOUND .Code is available at https://github.com/guoyihonggyh/Off-Dynamics-Reinforcement-Learning-via-Domain-Adaptation-and-Reward-Augmented-Imitation.C. Training the classifier p(trg|s t , a t ) and p(trg|s t , a t , s t+1 ) The two classifiers are parameterized bu θ SA and θ SAS . To update the two classifiers, we sample one mini-batch of data from the source replay buffer D ζ src and the target replay buffer D ζ src respectively. Imbalanced data is considered here as each time we sample the same amount of data from the source and target domain buffer. Then, the parameters are learned by minimizing the standard cross-entropy loss: Thus, θ = (θ SAS , θ SA ) is obtained from: where d is the done signal, and the target is given by:L SAS = -E D ζ src [θ = argmin θ L CE (D ζ src , D ζ trg ) = argmin θ [L SAS + L SA ] C.2y(s t , a t , d) = r + γ(1 -d) min j=1,2 Q trg,j (s t+1 , a ′ ) -α log π(a ′ |s t+1 ) , a ′ ∼ π(a|s t+1 ).The actor loss is:max π E a∼π p trg (s t+1 |s t , a t ) p src (s t+1 |s t , a t ) [Q π (s, a) -α log π(a|s)] .DAIL In DARAIL, the policy is optimized with the reward estimator R AE with the true reward from the source domain. We also want to compare the vanilla imitation learning with importance weight. The objective is:min ζ max Dω E psrc,ζ t ρ(st, st+1) log Dω(st, st+1) + E (s t ,s t+1 )∼τ src π DARC t log(1 -Dω(st, st+1)) , (C.3)Then, following the Eq.(C.3), the objective of policy optimization without the reward estimator is:max ζ E psrc,ζ t -ρ(s t , s t+1 ) log D ω (s t , s t+1 ) . (C.4)We can view it as a reduced version of our proposed method, which uses the reward function provided by the discriminator and importance weight.MBPO [19] . MBPO is a model-based RL method. We train the MBPO in the source domain and deploy it to the target domain.MATL [20] . MATL modified the reward on both the source and target domains and aligned the trajectories on both domains. Unlike our method, they need access to the reward from the target domain.GARAT [10] GARAT is a grounded action transformation approach that simulates target transitions (s t , a t , s t+1 , r) in the source domain with modified action, where the modified action is learned from imitation learning.As discussed, we use the broken with probability for Ant and Walker2d. The dynamics shift created by freezing one action varies across environments. For instance, in the Ant robot, the 0-index controls the rotor between the torso and front left hip, while in the HalfCheetah, the 0-index controls the back thigh rotor. So, the broken Ant experiences a larger shift than the broken HalfCheetah if we break the 0-index for both environments. Also, the broken environment in Walker2d and Ant creates such a large dynamics shift that it is overly difficult to adapt from the source domain, i.e., DARC cannot obtain the optimal reward in the source domain. We then introduce the broken with probability p f to better control the magnitude of dynamics shift. Broken with probability p f means the 0-index action is frozen with probability p f and follows the commanded torque with probability 1 -p f . In Reacher and HalfCheetah, the source environment is broken with probability 1. Ant and Walker2d's source domain is broken with a probability of 0.8.Figure 5 shows the performance of DARC in Ant and Walker2d under different broken probability p f in the source domain. We can observe that when p f = 1.0, the performance degradation of evaluating in the target domain is larger than the p f = 0.8 case. Also, when p f = 1.0, the DARC evaluation performance in the target domain is close to 0. Moreover, we notice that in the p f = 1.0 case, DARC training performance in the source domain receives a much lower reward than the p f = 0.8 case. However, we want to mimic the DARC behavior in the imitation learning, so we want DARC to be able to receive optimal reward in the source domain. Thus, for the Ant and Walker2d environment, we choose p f = 0.8 for the source domain. We show the training curve of DARAIL and baselines in different environments under the broken source environment setting in Figure 6 corresponding to the result in Table 3 . We also show the training curve of modifying the configuration in Figure 7 and 8 . Figure 10 : Experiments of DARC and DARAIL on the intact source and broken target setting. We observe that the DARC does not have significant performance degradation. Also, we show that DARAIL can perform similarly to DARC in this setting.Figure 9 shows the performance of DARC trained in the source and evaluated in the target domain under broken source environment setting. The training reward is the reward obtained in the source domain, i.e. E πDARC,p src [ t r(s t , a t )] and the evaluation is the reward deployed in the target domain, i.e. E πDARC,p trg [ t r(s t , a t )]. We observe the performance degradation in the figure 9 . Empirically, we notice that the DARC policy performance in the source domain, E πDARC,p src [ t r(s t , a t )], is close to the optimal reward in the target domain which matches with the DARC objective that DARC can generate target optimal trajectories in the source domain. However, deploying it to the target domain will result in performance degradation and a suboptimal reward due to the dynamics shift.We show the performance of DARAIL in the intact source and broken target environment setting in Figure 10 (the setting in DARC paper [3] ). We observe that our method outperforms the DARC reward in the target domain, E πDARC,p trg [ t r(s t , a t )]. Also, we see that the performance of DARC in the source domain and target domain are very similar. Compared with the performance gap when the source environment is broken in Figure 9 . As discussed, DARC works well when the assumption that the target optimal policy performs well in the source domain is satisfied. In the broken target setting, the target optimal policy can perform the same in the source domain.Further, empirically, in the broken target setting, the DARC policy learns a near 0 value for the broken joint, which guarantees that the policy can generate similar trajectories in the two domains. Also, maximizing the adjusted cumulative reward in the source domain with a policy with a near 0 value for the broken joint is equivalent to maximizing the cumulative reward in the target domain. Thus, DARC perfectly suits the broken target setting. However, in the broken source setting and other more general dynamics shift cases, the target optimal policy might not perform well in the source domain. For example, in the broken source setting, the target optimal policy will perform poorly in the source domain as it loses one joint in the source domain. Another way to understand why DARC fails is that it learns an arbitrary value for the broken joint, which becomes detrimental in the target domain. However, this is just an artifact of the particular setting. As we discussed above, the intrinsic reason that DARC fails is the violation of the assumption.In Figure 11 , We compare our DARAIL, which uses DARC trajectories in the source domain as expert demonstrations and mimicking source optimal trajectories regardless of the target domain.C.8 Access to the target domain data compared to DARC.Both DARC and DARAIL require some limited access to the target rollouts. In DARAIL, the imitation learning step only rolls out data from the target domain every 100 steps of the source domain rollouts, which is 1% of the source domain rollouts. We claim that more target domain rollouts will not improve DARC's performance due to its suboptimality, and DARAIL is better not because of having more target domain rollouts. We verify it by comparing DARC and DARAIL with the same amount of rollouts from the target domain in the broken source environment setting psrc(st,st+1) for the importance sampling method and DARAIL. Here, we compare the per-step importance weight with the cumulative n-step importance weight, which is the multiplication of the weight before time step t:ρ n (s t , s t+1 ) = t i=t-n p trg (s i+1 |s i , a i ) p src (s i+1 |s i , a i ) .Note that here, the importance weight is the multiplication of the last n steps weight instead of the multiplication from i = 0 to i = t. Because the cumulative importance weight might have a NaN value due to the product. Thus, the optimization step for the imitation learning of DARAIL is as follows:max ζ E psrc,ζ t ρ n (s t , s t+1 )r(s t , s t+1 ) -(1 -ρ n (s t , s t+1 )) log D ω (s t , s t+1 ) .Similarly, the objective of IS-R is:max π E psrc,π t ρ n (s t , s t+1 )r(s t , s t+1 ) .We compare the per-step importance weight and the cumulative n-step importance weight on DARAIL and IS-R. Specifically, we consider n = [10, 50] for HalfCheetah and Walker2d, respectively. We show the results of DARAIL in Figure 12 and the results of IS-R in Figure 13 . We see that the cumulative importance weight doesn't perform well on both methods and environments. In HalfCheetah, we can observe that the 10-step cumulative importance weight performs better than the 50-step one. And similar patterns appear in the Walker2d. Thus, we can conclude that per-step importance weight will have a lower variance and be more favorable in our experiment. In imitation learning, we alternatively update the generator and discriminator. In practice, we normally update the generator several steps and then update the discriminator once. The update steps, updating the discriminator every how many training steps, is a hyperparameter and is important in GAN training. The smaller the update steps are, the higher the update frequencies are. We tune the update steps and show the result of it in different environments. The best discriminator update step in HalfCheetah, Walker2d, and Reacher are 50, 50, and 1000, respectively. We varied the discriminator update steps in HalfCheetah and Walker2d in [10, 50, 500, 1000] steps, and the update steps in Reacher are [50, 100, 1000, 2000] steps. Figure 14 shows the effects of different discriminator update steps in the final performance. As we can see, for all three environments, a smaller update step (higher update frequency) is preferred as it can learn a better reward estimation. However, as we noticed, for example, for HalfCheetah and Walker2d, when the update step is 50, decreasing it to 10 will not further improve the performance.Increase the weight on the modified reward of DARC.We tested DARC algorithm with modified reward r(s t , a t ) + η∆(s t , a t , s t+1 ) with η > 1 instead of η = 1. And the η = 1 is derived from the distribution matching objective in Eq.(3.3). We show the results in Figure 15 under the broken source environment setting. We can see that increasing η will not increase the DARC performance in the target domain but will hurt the performance of DARC in the target domain.For a fair comparison, we tune the parameters of baselines and our method. The hidden layers of the policy and value network are [256, 256] for the HalfCheetah, Ant, and Walker2d and [64,64] for Reacher. And the hidden layer of the two classifiers is [64] for the HalfCheetah, Ant, and Walker2d and [32] for Reacher. The batch size is set to be 256. We regularize the state by adding the running average of the state. We fairly tune the learning rate from [3e -4, 1e -4, 5e -5, 1e -5]. For those methods that require the importance weight ρ, we tune the update steps of the two classifiers trained to obtain the importance weight from [10, 50, 100] . We also add Gaussian noise ϵ ∼ N (0, 1) to the • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: In the conclusion, we briefly mentioned that our method avoids directly training a policy in a high-risk environment in safety-critical tasks. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper proposes an RL method for off-dynamics RL (adapting to new environments) by combining reward augmentation (as done in prior work) with imitation (novel contribution). The reviewers generally lean towards accepting the paper: scores = 7/6/5/5. The reviewers appreciated the importance of the problem, the clear motivation, thorough experiments, and the theoretical results. Several reviewers had writing suggestions, which the authors have promised to make. During the rebuttal period, the authors and reviewers had a thorough discussion, which seems to have resolved most of the concerns the reviewers initially had about the paper (e.g., with additional experiments). Taking this together, I recommend that the paper be accepted.

Reviews: review: paper_summary: paper_summary: The authors introduced the approach Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) for off-dynamics RL. The work aim at generating the same trajectories in the target domain as expert trajectories learned via DARC in the source domain. With GAIL-style framework and reward augmentation, DARAIL reached SOTA in bench mark off-dynamiacs environments. DRAIL does not assume the learned expert policy in the source domain to be close to the optimal policy in the target domain, and the authors are able to give a theoretical error bound of the method according to the dynamics shift scale.
strength_weakness: strength_weakness: 1. Typo in Line #81, expectation should be over p_trg
2. Typo in Appendix C.2 Line #537 (C.1)
3. Typo in Appendix C.2 Line #540, #546
4. There is no comparison experiements of directly using (s,s’) from the expert trejectories in the source domain for DARAIL.
5. Freezing the 0-index action is not a good way to validate the effectiveness of the method on dynamics shift. Experiemnts on ,ore diverse dynamics shift environments are needed.
questions: questions: How would DARC trajectories quality influence the overall imitation learning performance in the target domain?
limitations: limitations: Did not test on more complicated dynamics shifts.
paper_summary: paper_summary: The paper considers the off-dynamic RL setting and identifies that existing approach, DARC, fails to obtain the true optimal policy in the target MDP. Leveraging ideas from imitation learning literature, the paper proposes a learning objective that takes into the account the dynamic shift between the source and target MDPs. The objective is theoretically sound and the conducted experiments show that the proposed method, DARAIL, is able to account for severe dynamic shift in MuJoCo environments.
strength_weakness: strength_weakness: - The notation can be defined more clearly.
	- On page 3, line 106: Is $\tau$ a trajectory sequence of state-action pairs, or just sequence of states?
		- What generates this?
	- On page 3, Eq (3.2): Is $\tau_{\pi^*}^{trg}$ Is this trajectory generated by $\pi^*$ in the target domain?
- It will be preferable if the paper can provide explanations on why DARC fails to address for the dynamic shift.
questions: questions: **Questions**
- On page 4, lines 120-122: Is this always true, or just may? If I understand correctly since the dynamics are "unknown" it may recover the wrong target policy (i.e. $\pi^* \neq \pi^{trg}$)?
- DARC does not need any online learning while DARAIL does correct? I believe this should be highlighted if that is the case.
- On page 4, lines 151-152: This shared reward is an assumption correct? What if this is not true either?

**Possible typos**
- Page 2, line 128: Domain
limitations: limitations: **Limitations**
- The experiments are purely in MuJoCo so I am curious what can happen in other domain.
paper_summary: paper_summary: This paper provides solution to a specific kind of domain adaptation, where the source action space is a subset of the target action space. In this setup, the source policy might fail in the target domain, because the policy would output arbitrary values for the action dimensions only active in the target domain, but not in source. To resolve this, the paper disentangles the effect of actions, and considers the source domain's policy's state trajectories as the optimal trajectories to imitate in the target domain. By applying imitation learning from observation, they can learn a target policy to reproduce the state transitions from the source policy, but using the action space from the target domain. One trivial solution to this is that the target policy would learn the source policy's actions, just with a 0 output (or a value that leads to no effect) for the extra dimensions in the target domain. The paper outperforms the prior work DARC on its specialized environment setup, but unlikely to be a general method for other off-dynamics setups.
strength_weakness: strength_weakness: ## Evaluation on a very specific kind of dynamics change
The major issue with this work is its limited evaluation where the only kind of off-dynamics shift is freezing the 0-index value to 0 in the action in the source domain. Ideally, if the action support of source is a subset of the target domain, then the same policy should be perfectly fine in the target domain. Yet, this paper shows that the same policy underperforms in the target domain. The only reason why this might be happening is because the DARC policy learns an arbitrary value for the 0-th dimension of action torque, which is inconsequential in the source domain, however, becomes detrimental in the target domain. The paper never discusses this obvious reason for this failure, and the only thing a method needs to do is to learn to always output 0 for the 0th-index, for it to work in the target domain.

DARAIL is specifically constructed so that it learns to output 0 and works well for this artificial environment setup, because it **assumes the source trajectory to be the optimal trajectory in the target domain**. But this is a major assumption not made in the original DARC paper. In fact, I would argue (and I am open to change my mind on this if proven otherwise) that DARAIL does not work in the off-dynamics settings considered in DARC paper. For instance, consider ant which is crippled in the target domain but not the source domain (from DARC paper). Considering the source ant's trajectories as optimal could be detrimental for the target crippled ant, because most of the state transitions $(s_t, s_{t+1})$ in source domain are not even possible in the target domain. The crippled ant requires a different optimal policy to be learned, which cannot be obtained by just imitating the source state trajectories, which DARAIL would do.

In order to show that DARAIL is actually a general solution to off-dynamics RL, the paper should include experiments on:
- All the same setups provided in DARC paper.
- At least one more example of off-dynamics RL other than just freezing the 0-index of action.

## Paper Presentation
The paper is often written in a confusing manner and there are several quality issues in presentation. The following need to be corrected in writing:
- Figure 6 (b) for Ant and 6 (d) for Walker are the exact same training curves. Please fix this.
- The paper never discusses why DARC "fails" is because of the 0-th action dimension being an arbitrary output in the target domain, because it was never trained to be 0 during training.
- L2-3: "performance degradation especially when the source domain is less comprehensive than the target domain": "less comprehensive" is vague.
- In L23-25, the paper mentions that "in domains such as medical treatment [1] and autonomous driving [2], we cannot interact with the environment freely as the errors are too costly". However, in Algorithm 1 L7, it assumes access to the target environment for rolling out.
- L41-42: "the source domain has less action support, which is normally a harder off-dynamics RL problem." Why is this a harder problem? If anything, target domain having a less action support would mean that the agent needs to adapt to more adverse off-dynamics conditions.
- L64-65: "DARAIL relaxes the assumption made in previous works that the optimal policy will receive a similar reward in both domains." — What does this mean, because L78 says that both the source and target domains have the same reward function? In fact, DARAIL considers the source trajectories as optimal trajectories for the target domain.
- What does it mean for the two domains to have the same reward function, when the dynamics are different? Does the paper assume the reward function is of the form R(s) only and not R(s, a, s')? Any assumptions made in the problem setup should be clearly written.
- In Eq 3.1 and 3.2, the product term $\Pi_t$ should include the last terms inside the bracket.
- L110-111: This cannot be right. The optimal policy should be proportional to the exponential of the cumulative returns, not reward at time step t.
- Figure 1 (a): It is unclear how this figure suggests that there is any **performance degradation**. Since the dynamics are different, it is possible that the optimal reward in the target domain could be just different from the optimal reward in the source domain — comparing these two does not make sense. It only works in this paper because of the artificial formulation that the source action space is a subset of the target action space. The correct comparison would be evaluating $pi_{DARC}$ in the target domain against some $\pi_{optimal}$ for the target domain, both in the target domain. The training reward obtained in the source domain does not tell us anything for general off-dynamics setups.
- L151-166: This paragraph is hard to follow.
- Figure 7: Some baselines are not trained for the same number of steps.

## Method is overly complicated for the problem setup
The paper makes a questionable assumption that when $\pi_{DARC}$ is optimal for source domain, the trajectories it obtains are also optimal in the target domain and thus, we should achieve similar state trajectories in the target domain. While this is a very restricted assumption, let's say for the sake of the argument, this assumption makes sense for certain useful setups. Even then, why learn $\pi_{DARC}$ using any influence from the target trajectories at all? Why not just learn $\pi_{src}$ from the src reward only, while disregarding the target trajectories and not applying DARC at all. And then we can transfer that policy using Gaifo to the target environment. What is the idea of doing DARC at all, when anyway the source policy is to going to be used for imitation learning from observations in the target domain? If DARC on src is really important, it should be experimentally validated with convincing arguments.
questions: questions: I have listed suggestions to improve in the weakness section above. There are a lot of changes in the paper presentation that are necessary, but my major concern would still be about the limited applicability of the proposed problem setup and the method not even best suited for that proposed problem.
limitations: limitations: The paper mentions a couple of limitations in the appendix section, not in the main paper. The paper does not discuss several limitations of their problem setup's applicability, their method's potential fallacies on the original DARC environments, and their limiting assumption of source trajectories being optimal for the target domain.
paper_summary: paper_summary: This work introduces an improvement over the previous method (DARC) with theoretical and experimental contributions for solving the off-dynamics transfer learning problem.
strength_weakness: strength_weakness: - It's not clear how many samples are used as rollout in the target domain. If it is a non-trivial amount, then a baseline should be to run RL in the target domain with this sample budget.
- Similarly, DARC should also be allowed to have these additional samples in the target domain for proper comparison.
- Finally, the presentation quality poses a bit of a weakness. It is a bit difficult to read/analyze at times (see next section for suggestions to improve)
- Figure 3 does not appear to be a significant performance boost, can you comment on the purported gain?
questions: questions: Questions about the paper:
In the appendix you show that lower discriminator update steps are better, stopping at 50. Can you explain why lower might be better; and why not decrease even further?
- You state a few times that you do not collect reward data from the target domain. Isn't this trivial since the reward function is identical to that of the source?


Various (some minor) presentation issues and references:
- suggest to abbreviate as "GAIL-O" or similar. Readers will likely know GAIL and thus understand the connection more quickly
- In main contributions, the first bullet point is quite unclear; can you re-word/shorten? Similarly, I believe you can remove all but the last sentence in bullet point 2.
- "works" -> "work" throughout
- Use of discounting: You use SAC with a discount factor, but $\gamma$ is missing from the main text and appendix in many equations. Can you include / comment on this? It appears the theory is only relevant for $\gamma=1$ which is okay but should be stated upfront.
- Relatedly, it may be beneficial to cite in Sect. 3.1 e.g. https://arxiv.org/abs/1805.00909 ;  https://arxiv.org/pdf/2005.01643 
- Regarding the dynamics-shift problem, can you also comment on the relationship to e.g. https://proceedings.neurips.cc/paper_files/paper/2023/file/67dd6a41bf9539cffc0fc0165e4d0616-Paper-Conference.pdf ; https://ojs.aaai.org/index.php/AAAI/article/view/25817 ; https://proceedings.mlr.press/v155/pertsch21a/pertsch21a.pdf 
- L80 "number of data" -> "amount of data"
- L84: "to mimic **an** expert"
- (nitpicking) L90: discussion of observations vs. states does not sound quite right; I believe the true distinction is in the POMDP framework
- Eq. 3.1 remove the extra parentheses around product.
- Eq. 3.3 Do you still need a min on the RHS? (I might be missing something, please explain if so!)
- the value of $c$ may be nice to see explicitly so that we can see independence on the dynamics and policy.
- L128: "Domian" -> "Domain"
- L135 "while we do not have much access to the target domain." can you be more specific?
- L150: "followed" -> "follows"
- L172: "Baye's" -> "Bayes'"
- Algorithm 1; maybe highlight which parts of the algorithm are distinct from DARC?
- Theorem 4.1: I believe it would be helpful to either (a) give a more formal version of the theorem with all definitions (define H, D_omega, is $\gamma$ missing?) or (b) make it an "informal" theorem, with the full result in the appendix.
- Which version of the environments are you using?
- L304: "DARAIL's ??? on Different..." is there a missing word (Effect)?
- Figures: Can you please include a shaded region for the DARC eval/training reward lines? 
- Figure 3: DARRIL- -> DARAIL?



----
Overall, I believe the paper needs a significant clean-up of writing and additional experiments and discussion to merit a higher score.
limitations: limitations: As stated above, two important limitations the authors can further comment on are (a) use of samples in the target domain and (b) gain in performance.

Comments: comment: Thank you very much for your reply! 

Best,

Authors
5, **Paper presentation.** 

**Typo.** Thanks for the suggestion on the paper writing. We will fix the typo in writing, including the Figure, $\prod_t$ in Eq 3.1/3.2. Also, the optimal policy is proportional to the exponential of the cumulative rewards, not reward at time step t in Lines 110-111.

**More explanation to Line 151-166 regarding the reward estimator $R_{AE}$.** The imitation learning step iteratively trains a discriminator (maximizing Eq 3.5) and then updates policy with the signal provided by the discriminator, which is $-log_{D_{\omega}}(s_t,s_{t+1})$. We can view this as a ‘local reward estimator,’ and the policy is updated with $-log_{D_{\omega}}(s_t,s_{t+1})$ in imitation learning instead of the reward $r(s_t,a_t)$. However, this estimation can be biased and inaccurate in generative adversarial training, and the problem is even more severe due to the off-dynamics shift. Similarly, in the off-policy evaluation (OPE) of offline bandit and RL, this bias and inaccurate reward estimation exist. And doubly robust estimator is employed in this case to correct the reward estimation with importance weight term. Motivated by the doubly robust estimator, we propose a surrogate reward function that uses the source domain reward $r_{src}(s_t,a_t)$ (as the reward function is the same across the domains) and the importance weight between the dynamics to correct the local reward estimator $-log_{D_{\omega}}(s_t,s_{t+1})$, which is similar to the doubly robust estimator that uses $\rho (r-\hat{r})$ to correct the reward estimator $\hat{r}$. Here, the $\rho$ is the importance weight. 

We believe we have covered all the questions regarding paper presentations. If there is still more clarification needed, please let us know. Thank you again for the suggestions.
1, **Qualitative reasoning for why DARC fails in the gravity-change setting, but DARAIL works perfectly fine?** As mentioned in the previous general response, DARC works when the performance of the target optimal policy is similar in the source domain. This is hardly true when we have a different gravity in general. For example, in simple cases like Pendulum and InvertedPendulum, gravity parameters directly change the angle of the pole, the velocity of the cart, etc. Thus, it’s possible that the target optimal policy can lead to a different angle and velocity, which could be more likely to fall for the pole in the source domain. Intuitively, in this gravity-changing setting, applying the DARC policy to the target domain might generate very different trajectories in the target domain than the source ones, causing performance degradation. For example, in the InvertedPendulum, we run experiments modifying the gravity coefficient from 1.0 to 10. DARC receives optimal rewards in the source domain, but only 40\% of the source domain rewards in the target domain. In this setting, the reward is defined as whether the pole stays on the cart at each step and the trajectory ends once the pole falls. This means that the length of DARC target trajectories is only 40\% of DARC source trajectories.

**DARAIL works in this setting.** DARAIL works well in this setting because we propose using imitation learning to transfer these trajectories to the target domain instead of directly deploying DARC to the target domain. By doing so, we do not require the assumption that the dynamic of the target optimal policy performs similarly in the source domain. Though DARAIL utilizes DARC trajectories in imitation learning, it doesn’t mimic the DARC policy directly but mimics the DARC trajectories in the state space in the source domain, which avoids deploying DARC directly to the target domain. 

2, **DARC significantly underperforms for HalfCheetah but works almost well for Reacher.** First, the experiment results are averaged over 5 runs. Second, in Reacher, changing the gravity may not create a significant dynamic shift compared to the HalfCheetah, though the coefficient of gravity is modified from 1.0 to 0.5. Besides gravity, we also noticed that modifying the “friction” of the Reacher configuration file from 1.0 to 0.5 or 2.0 will not cause very large performance degradation. How they affect the performance degradation is determined by how the parameters like gravity and friction affect the dynamics model. For tasks that depend more on gravity, like HalfCheetah, the performance degradation is more obvious. We will have more experiments in different environments, dynamic shifts, and scales of the shifts, including gravity, friction, and density, in our revision. 

3, **Mimicking the source optimal policy.** Our experiment is exactly what you describe that using the unmodified reward $𝑟_{unmodified}=𝑟(𝑠_𝑡,𝑎_𝑡) $ in Step 2: [call DARC]. 

4, **Modifying DARC $\Delta r$’s scale.**  Thanks for the suggestion. As mentioned, DARC training constrains the DARC policy to behave similarly in the two domains, and the  $\Delta r$ in the modified reward can be regarded as the constraint or the regularization term during the training. Thus, increasing the $\alpha$ will give a stronger constraint for policy optimization, making DARC behave more similarly in both domains. However, that doesn’t guarantee that DARC policy will perform well in the target domain. Moreover, increasing the $\alpha$ (stronger regularization) might affect the training to focus more on the constraint instead of maximizing the reward. We further run experiments on HalfCheetah with the broken source, the intact target environment, and the gravity-changing setting (from 1.0 to 0.5) to justify this. The following table shows the results of DARC in HalfCheetah with different $\alpha$. As we increase $\alpha$ from 1.0 to 2.0, we observe that the DARC reward in the source and target are getting closer for both cases. However, the training has a strong oscillation due to the regularization. Due to the regularization, though DARC evaluation can benefit from increasing $\alpha$, the DARC training performance will be affected to be close to the DARC evaluation performance. In this case, DARAIL still outperforms DARC with $\alpha = 2$. We will experiment more and add this baseline on the different $\alpha$ in our revision.

|            |DARC Training, alpha = 2 | DARC Evaluation, alpha = 2| DARC Training, alpha = 1 | DARC Evaluation, alpha = 1|  DARAIL|
|----|-----------|---------------------------------------------|-----------------|--|-----------------|
|Changing Gravity                | 2438   | 2307 | 5828   | 1544  | 5818|
|Broken Environment           | 6171  | 5955  | 6995 |  4133  | 7067|
Thank you very much for your positive feedback! We will revise our paper according to your constructive reviews.

Best,

Authors
Thank you to the authors for responding to my questions. I believe you have somewhat addressed my concerns and hopefully the camera-ready version can include some of this discussion. As stated before, the paper will benefit significantly from a careful rewriting, which I hope the authors will do. I've accordingly raised my score to 5.
Thank you for the response, regarding (3) you have addressed my question. I will keep the score as is.
Thank you for adding the new experiments on off-dynamics settings and DARC settings. The results of the intact source and broken target environment (DARC) are convincing to me, and they make sense to me now. The DARC policy learned in the source is on the modified reward and thus will tend to imitate the target data, thus generating trajectories that resemble optimal state sequences in the target domain. So, applying DARAIL should not deteriorate your starting point of DARC. This also explains why using just an optimal source policy is not enough, and you need to start with a DARC policy. I am raising my score to 5 and would be willing to increase further if the following questions are answered.

1. I appreciate you confirming that DARC fails in your original setting because it learns an arbitrary value of zero-index. Can you provide similar qualitative reasoning for why DARC fails in the gravity-change setting but DARAIL works perfectly fine? I understand that the underlying reason is that the assumption of the DARC paper might not hold, but can you discuss how exactly that turns out in the half-cheetah/reacher environments? And why does DARAIL (which also uses a DARC-like policy) not suffer from the same issues and perform so well?

2. In the gravity experiment, DARC significantly underperforms for HalfCheetah (1544 v/s 5818) but works almost as well for Reacher.
- What is the reasoning here, and how many seeds are these results?
- The poor performance of DARC in HalfCheetah is quite extreme; the gap is even larger than the experiments in the original paper. This is impressive but also surprising. I don't understand why DARC would fail so significantly here.
- Actually, now that I think about it, does Reacher even get affected by the coefficient of gravity?

3. In the experiment on optimal source policy imitation, did you exactly run Algorithm 1, with the only change being that you replace Step 2: [Call DARC], you do not use the modified reward $r_{\text{modified}} = r(s_t, a_t) + \Delta r(s_t, a_t, s_{t+1})$, but use the environment reward only: $r_{\text{unmodified}} = r(s_t, a_t)$ ? If not, what is the "Mimic Source Optimal Policy" experiment?

4. If I understand correctly, DARAIL exploits some rollouts in the target environment to match the DARC's source trajectory distribution. This results in a DARAIL policy that is more suitable to be deployed in the target environment than DARC's learned policy. But, could this effect of making the effect of target "stronger" be done in another way, like learning a modified DARC policy with $r_{\text{modified}} = r(s_t, a_t) + \alpha \Delta r(s_t, a_t, s_{t+1})$, where $\alpha > 1$? I know the authors added a rebuttal experiment with more target rollouts to DARC. But that only makes the $\Delta r$ term more accurate. It does not increase its influence in the DARC's learned policy. If my understanding is correct here, then this simple solution should be a baseline in this paper. If the authors can try it out, that would be great. The implementation change is as easy as adding a coefficient and experimenting with values like 1.5, 2, 10, etc. — depending on the reward scale. But if not, I would be happy with an explanation of why this would not work. Why is DARAIL, which is much more complex and comes with additional requirements for online environment rollouts with DARC policy, necessary?

5. There are several points in the paper presentation that were not addressed (understandably due to the lack of space in rebuttal response). Some of these are important. Now that there is no word limit, can you make sure everything is replied to?