Title: Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation
Venue: NeurIPS 2024
Accepted: yes
Keywords: off-dynamics reinforcement learning

domain adaptation

imitation learning

dynamics mismatch
Abstract: Training a policy in a source domain for deployment in the target domain under a dynamics shift can be challenging, often resulting in performance degradation. Previous work tackles this challenge by training on the source domain with modified rewards derived by matching distributions between the source and the target optimal trajectories. However, pure modified rewards only ensure the behavior of the learned policy in the source domain resembles trajectories produced by the target optimal policies, which does not guarantee optimal performance when the learned policy is actually deployed to the target domain. In this work, we propose to utilize imitation learning to transfer the policy learned from the reward modification to the target domain so that the new policy can generate the same trajectories in the target domain. Our approach, Domain Adaptation and Reward Augmented Imitation Learning (DARAIL), utilizes the reward modification for domain adaptation and follows the general framework of generative adversarial imitation learning from observation (GAIfO) by applying a reward augmented estimator for the policy optimization step. Theoretically, we present an error bound for our method under a mild assumption regarding the dynamics shift to justify the motivation of our method. Empirically, our method outperforms the pure modified reward method without imitation learning and also outperforms other baselines in benchmark off-dynamics environments.

Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation

Abstract

Training a policy in a source domain for deployment in the target domain under a dynamics shift can be challenging, often resulting in performance degradation. Previous work tackles this challenge by training on the source domain with modified rewards derived by matching distributions between the source and the target optimal trajectories. However, pure modified rewards only ensure the behavior of the learned policy in the source domain resembles trajectories produced by the target optimal policies, which does not guarantee optimal performance when the learned policy is actually deployed to the target domain. In this work, we propose to utilize imitation learning to transfer the policy learned from the reward modification to the target domain so that the new policy can generate the same trajectories in the target domain. Our approach, Domain Adaptation and Reward Augmented Imitation Learning (DARAIL), utilizes the reward modification for domain adaptation and follows the general framework of generative adversarial imitation learning from observation (GAIfO) by applying a reward augmented estimator for the policy optimization step. Theoretically, we present an error bound for our method under a mild assumption regarding the dynamics shift to justify the motivation of our method. Empirically, our method outperforms the pure modified reward method without imitation learning and also outperforms other baselines in benchmark off-dynamics environments.

1 Introduction

The objective of reinforcement learning (RL) is to learn an optimal policy that maximizes rewards through interaction and observation of environmental feedback. However, in domains such as medical treatment [1] and autonomous driving [2], we cannot interact with the environment freely as the errors are too costly or the amount of access to the environment is limited. Instead, we might have access to a simpler or similar source domain. This requires domain adaptation in reinforcement learning. In this paper, we study a specific problem of domain adaptation in reinforcement learning (RL), where only the dynamics (transition probability) are different in two domains. This is called off-dynamics RL [3-5]. Specifically, we focus on a problem setting in which we have limited access to rollout data from the target domain, but we do not have access to the target domain reward, following the previous off-dynamics work [3-5].

Previous work on off-dynamics RL, such as Domain Adaptation with Rewards from Classifiers (DARC) [3] and [6, 5], focuses on training the policy in the source domain with a modified reward function that compensates for the dynamics differences. The reward modification is derived so that the distribution of the learning policy's experience in the source domain matches that of the optimal trajectories in the target domain. As a result, their experience in the source domain will

<latexi sh

1_b

64="vf+MOWCX

kD7

B/Zj9

qm3

yT0

>A

H

c

V

NS8

EJ

Ur

I

Q

o

Y

g

Gz

L

5u

w

Pp

K

nd2

R

F

<latexi sh

1_b

64="vf+MOWCX

kD7

B/Zj9

qm3

yT0

>A

H

c

V

NS8

EJ

Ur

I

Q

o

Y

g

Gz

L

5u

w

Pp

K

nd2

R

F

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

Reward

Training Reward

Evaluation Reward

Optimal Reward in Target

Source

Domain

Target

Domain

Discriminator

Reward

Augmentation

Discriminator Training

Generator

Training

(

s

t

, s

t

+1

)

Reward

Modification

Source

Domain

Roll out from source domain

to use as expert data

Calculate modified reward

Obtain state pairs from source

Discriminator training

Policy optimization

Policy

network

Policy

network

DARC Training

Importance Weights

(

s

t

, s

t

+1

)

Importance Weights

produce a trajectory distribution close to the target domain's optimal one. However, deploying the resulting policy in the target domain usually causes performance degradation compared to its training performance in the source domain. Figure 1 (a) shows the experiment result of DARC under a broken source environment setting, where the broken source environment means the value of 0-index in the action of the source domain is frozen to 0, and the target environment remains intact. Consequently, existing reward modification methods will only obtain a sub-optimal policy in the target domain. Details of DARC and its suboptimality in the target domain will be introduced in Section 3.1. More details about why DARC fails in more general dynamics shift cases are in Appendix C.6.

In this paper, we present an off-dynamics reinforcement learning algorithm described in Figure 1 (b). Our method, Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) consists of two components. Following previous work like DARC [3] on off-dynamics RL, we first obtain the source domain trajectories that resemble the target domain's optimal ones. We then transfer the policy's behavior from the source to the target domain through imitation learning from observation [7], which can mimic the policy's behavior from the state space.

In particular, we consider the dynamics shift in the framework of generative adversarial imitation from observation (GAIfo) [8], and propose a novel and practical reward estimator called the reward augmented estimator ( R AE ) for the policy optimization step in imitation learning.

Our contributions can be summarized as follows:

We propose the Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) algorithm by transferring the learned policy of reward modification approaches from the source domain to the target domain via mimicking state-space trajectories in the source domain. We propose reward augmented estimator ( R AE ) to leverage the reward from the source domain to stabilize the learning.

We recognize limitations in the existing DARC algorithm and off-dynamics reinforcement learning algorithms with similar reward modification, which is directly deploying the learned policy to the target domain results in significant performance degradation. Our proposed algorithm mitigates this issue with an imitation learning component that transfers DARC policy to the target.

We introduce an error bound for DARAIL that relaxes the assumption made in previous works that the optimal policy will receive a similar reward in both domains. Specifically, with our imitation

learning from the observation component, we can show the convergence of DARAIL with a mild assumption on the magnitude of the dynamics shift.

We conducted experiments on four Mujoco environments, namely, HalfCheetah , Ant , Walker2d , and Reacher on modified gravity/density configurations and broken action environments. A comparative analysis between DARAIL and baseline methods is performed, demonstrating the effectiveness of our approach. Our method exhibits superior performance compared to the pure modified reward method without imitation learning and outperforms other baselines in these environments. Code is available at https://github.com/guoyihonggyh/Off-Dynamics-Reinforcement-Learning-via-DomainAdaptation-and-Reward-Augmented-Imitation.

2 Backgrounds

̸

Off-dynamics reinforcement learning We consider two Markov Decision Processes (MDPs): one is the source domain M src, defined by ( S , A , R , p src , γ ) , and the other one is the target domain M trg , defined by ( S , A , R , p trg , γ ) . The difference between them is the dynamics p , also known as transition probability, i.e., p src = p trg or p src ( s t +1 | s t , a t ) = p trg ( s t +1 | s t , a t ) . In our paper, we experiment with two types of dynamics shift: 1) broken environment [3], in which the 0-th index value is set to be 0 in action, and 2) modifying the gravity/density setting of the target environment [9]. The source and the target domain share the same reward function, i.e., r src ( s t , s t +1 ) = r trg ( s t , s t +1 ) . All other settings, including state space S , action space A , and the discounting factor γ , are the same. We will use γ = 1 in the derivation and analysis in our paper.

̸

We aim to learn a policy ζ ( a | s ) using interaction from the source domain together with a small amount of data from the target domain ( s t , a t , s t +1 ) trg to maximize the expected discounted sum of reward E ζ,p trg [ ∑ t γ t r ( s t , a t )] in the target domain. Note that we assume we only have limited access to the target domain transition, namely ( s t , a t , s t +1 ) trg, in the whole process and we do not utilize the target domain reward.

Imitation learning (from Observation) Imitation Learning (IL) trains a policy to mimic an expert policy π E with expert demonstration { ( s 0 , a 0 ) , ( s 1 , a 1 ) , ... } or { ( s 0 , s 1 ) , ( s 1 , s 2 ) , ... } . Generative adversarial imitation learning (GAIL) [7] uses an objective similar to Generative adversarial networks (GANs) that minimizes the distribution generated by the policy and the expert demonstration. It alternatively trains a discriminator D ω and a policy π θ to solve the min-max problem:

where s ′ is the next state and H ( π θ ) is the entropy of the policy π θ . Note that in our problem, we mimic the state-only expert demonstrations { ( s 0 , s 1 ) , ( s 1 , s 2 ) , ... } instead of the expert's actions. This setting is also called imitation learning from observation [8]. We will further discuss why we use state observation instead of action in section 3.2. D ω is the classifier that discriminates whether the state pair is from the expert π E or generated by the policy π θ . Then, the policy is trained with the RL algorithm using reward estimation -log D ω ( s, s ′ ) as the reward. The optimization of the Eq. (2.1) involves alternatively training the policy and the discriminator.

3 Off-dynamics RL via Domain Adaptation and Reward Augmented Imitation Learning

In this section, we present our algorithm, DARAIL, under the off-dynamics RL problem setting. First, we introduce DARC [3] in Section 3.1, which provides the distribution of target optimal trajectories in the source domain to mimic. Then, in Section 3.2, we introduce the imitation learning component through which we utilize the trajectories provided by DARC and transfer the DARC policy to the target domain. We aim to learn a policy that generates the same distribution of trajectories in the target domain as the DARC trajectories in the source domain.

3.1 Off-dynamics RL via Modified Reward

DARC is proposed to solve the off-dynamics RL through a modified reward that compensates for the dynamics shift [3]. Here, we first introduce DARC and its drawbacks. DARC seeks to match the policy's experiences in the source domain and optimal trajectories in the target domain. We

define τ = { ( s 1 , a 1 ) , ( s 2 , a 2 ) , ..., ( s t , a t ) , ... } as a trajectory. We use τ src π θ to represent the trajectories generated by π θ in the source domain. The policy's distribution over trajectories in the source domain is defined as:

Let π ∗ = argmax π E π,p trg [ ∑ t r ( s t , a t )] be the policy maximizing the cumulative reward in the target domain. We use τ trg π ∗ to represent the trajectories generated by π ∗ in the target domain. Given the assumption that the optimal policy π ∗ in the target domain is proportional to the exponential reward, i.e., π ∗ ( a t | s t ) ∝ exp( ∑ t r ( s t , a t )) , the desired distribution over trajectories in the target domain is defined as:

DARC policy can be obtained by minimizing the reverse KL divergence of p ( τ trg π ∗ ) and q ( τ src π θ ) :

where ∆ r ( s t , a t , s t +1 ) := log p trg ( s t +1 | s t , a t ) -log p src ( s t +1 | s t , a t ) and c is a partition function of p ( τ trg π ∗ ) , which is independent of the dynamics and policy. The ∆ r ( s t , a t , s t +1 ) can be calculated through the following procedure: i), train two classifiers p ( trg | s t , a t ) and p ( trg | s t , a t , s t +1 ) with cross-entropy loss L CE ; ii), Use Bayes' rules to obtain the log ( p trg ( s t +1 | s t ,a t ) p src ( s t +1 | s t ,a t ) ) . Details are in Appendix C.1. Eq. (3.3) shows that π DARC can be obtained via maximum entropy algorithm with a modified reward r modified = r ( s t , a t ) + ∆ r ( s t , a t , s t +1 ) at every step.

However, DARC matches the distribution of τ trg π ∗ and τ src π DARC . As the dynamics shift exists, π DARC will not recover the optimal policy π ∗ , and deploying the DARC in the target domain will usually suffer from performance degradation due to the dynamics shift, as shown in Figure 1(a) and Figure 9 in Appendix. However, in the source domain τ src π DARC resembles those optimal trajectories in the target domain. Given the property of τ src π DARC , we propose to use imitation learning from observation with τ src π DARC as expert demonstrations to transfer DARC to the target domain. The new policy in the target domain should behave similarly (generate similar trajectories) as DARC in the source domain.

3.2 Imitation Learning from Observation with Reward Augmentation

In this section, we present the Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) method, which mitigates the problem of DARC via imitation learning from observation. As described in Section 3.1, τ src π DARC resembles the target optimal trajectories, and we want to transfer DARC's behavior to the target domain. A natural way to tackle it is utilizing imitation learning to mimic the expert demonstration τ src π DARC . Following [7, 8], the objective can be formulated as:

where D ω is the discriminator in the generative adversarial imitation learning and ζ is the policy to be learned in the target domain. In the objective function Eq. (3.4), the ( s t , s t +1 ) pairs are from the target domain, while we do not have much access to the target domain. Alternatively, we can use the ( s t , s t +1 ) pairs from the source domain and re-weight the transition with the importance sampling method to account for the dynamics shift. The objective with data rolled out from the source domain, and the importance sampling is as follows:

where ρ ( s t , s t +1 ) = p trg ( s t +1 | s t ,a t ) p src ( s t +1 | s t ,a t ) is the importance weight. Note that we do the generative adversarial imitation learning from only state observations ( GAILfo ) with ( s t , s t +1 ) [9-11] instead of ( s t , a t ) . This is because we aim to learn a policy ζ to produce the same trajectory distributions in the target as the ones π DARC produces in the source domain, despite the dynamics shift, rather than mimicking the policy. Mimicking the ( s t , a t ) pairs will recover the same policy as DARC, and deploying it to the target domain will not recover the expert trajectories due to the dynamics shift.

This objective Eq. (3.5) can be interpreted as training the discriminator D ω to discriminate whether the ( s t , s t +1 ) generated by ζ in the target domain matches the distribution of DARC trajectories

in the source domain using data rolled out from the source domain with ζ and importance weight. Then, after the discriminator is fitted, the policy can be trained with the reward estimator R AE with model-free RL. The objective is:

where R AE is defined as follows:

Here the r src ( s t , s t +1 ) is the reward obtained from the source domain, which is the same as the reward from the source domain, i.e. r trg ( s t , s t +1 ) . In imitation learning, the -log D ω ( s t , s t +1 ) can be viewed as a local reward function for the policy optimization step and the objective is max ζ E p src ,ζ [ ∑ t -log D ω ( s t , s t +1 )] . So Eq.(3.5) can be viewed as learning a reward function for the training of ζ . However, as the dynamics shift exists, the estimation of the -log D ω ( s t , s t +1 ) could be biased, which is similar to the case in off-policy evaluation (OPE) [12-16] when training a reward estimation on biased data. As we have access to the source domain and can obtain the reward from the rollout, we are motivated to use both the reward estimation -log D ω ( s t , s t +1 ) and the ground truth reward in the source domain r src ( s t , s t +1 ) so that we could have a better reward estimation than -log D ω ( s t , s t +1 ) under dynamics shift. The R AE here can be viewed as using -log D ω ( s t , s t +1 ) as a base estimator of the reward and use r src ( s t , s t +1 ) and importance weight ρ ( s t , s t +1 ) to correct it. This correction idea is similar to the doubly robust estimator (DR) [12] in OPE. The DR estimator combines the reward estimation ˆ r and the importance-weighted difference between true reward r and ˆ r . Specifically, the DR method takes the reward estimation ˆ r as a base estimator and applies the importance weighting to the difference between true reward r and ˆ r , which is ρ ( r -ˆ r ) term, to correct the bias of the ˆ r , where ρ is the importance weight.

Algorithm 1 Domain Adaptation and Reward Augmented Imitation Learning (DARAIL)

1: Initialize: source and target environments M src and M trg; replay buffers for source and target transitions, ( D π DARC src , D ζ trg , D ζ src); initial parameters for the two classifiers θ = ( θ SA , θ SAS ) ; initial policy ( π DARC , ζ ) ; initial discriminator D ω , ratio r of experience from source vs. target, ratio k of update frequency of generator vs. discriminator.

Our Algorithm The DARAIL is shown in Algorithm 1, which consists of two steps: the first step, Line 2 in Algorithm 1, is the training of π DARC, and the second step is imitation learning with the reward estimator in Eq. (3.7). In Lines 6-8, we roll out the target domain transition ( s t , a t , s t +1 ) to calculate the importance weight. Here, we will not collect the target domain reward. In Lines 9-11, we update the discriminator based on Eq. (3.5). In Line 12, we train the two classifiers p ( trg | s t , a t ) and p ( trg | s t , a t , s t +1 ) with cross-entropy loss L CE and Bayes' rules similar to ∆ r ( s t , a t , s t +1 ) in DARC as mentioned in Section 3.1. The details are in Appendix C.1. Lastly, we calculate the R AE in Line 13 and update the generator (Soft Actor-Critic (SAC) [17]) with R AE in Line 14.

Note that in Lines 6-7, we roll out from the target domain, but the amount of it is significantly smaller than the source rollouts. In our experiments, we roll out from the target domain every 100 steps of

source domain rollouts, which is 1% of the source domain rollouts. Further, even though DARAIL requires more target domain rollouts than DARC as it is required to train DARC first and then perform the imitation learning step, the advantage of DARAIL does not solely come from the more target samples. Because, in DARC, increasing the training step or target domain rollouts will not further improve its performance due to its inherent suboptimality, which is shown in table 11 and 12 in Appendix with the same amount of target domain rollouts.

4 Theoretical Analysis of DARAIL

Let π ∗ = argmax π E π,p trg [ ∑ t r ( s t , a t )] be the optimal policy maximizing the cumulative reward in the target domain and ˆ ζ be the policy learned from DARAIL. Now, we provide an error bound for DARAIL. Details of the proof are deferred to Appendix B.

Theorem 4.1. Let m be the number of the expert demonstration and ˆ R ( m ) π = E σ [ sup D ∈D 1 m ∑ m i =1 σ i D ( s t , s t +1 ) ] be the empirical Rademacher complexity. Let B be the error bound of DARC in the source domain, i.e. E p src ,π ∗ DARC [ ∑ t r ( s t , a t ) + H [ a t | s t ]] -E p src ,π DARC [ ∑ t r ( s t , a t )] ≤ B and W be the upper bound of the importance weight, i.e. ρ ( s t , s t +1 ) ≤ W , ∀ ( s t , s t +1 ) . Let discriminator class D be a ∆ -bounded function, i.e. | D ω ( s t , s t +1 ) | ≤ ∆ given any ( s t , s t +1 ) . ∥ r ∥ D measures the richness of the discriminator to represent the ground truth reward as defined in Appendix B.2. d D is a defined neural network distance between the ( s t , s t +1 ) distributions generated by the π DARC and π ˆ ζ defined in Appendix B.1. Given the empirical training error of the imitation learning, i.e. d D (ˆ τ src π DARC , ˆ τ trg ˆ ζ ) -inf ζ d D (ˆ τ src π DARC , ˆ τ trg ζ ) ≤ ˆ ϵ , ∀ δ ∈ (0 , 1) , with probability at least 1 -δ , we have

Remark 4.2. Our error bound depends on (1) the DARC error bound in the source domain and (2) the imitation learning generalization error, where (2) is further decomposed into (2.1) approximation error and (2.2) estimation error. This bound demonstrates how the two important components in our proposed approach contribute to a good performance. Firstly, we would want a well-trained policy on the source to reduce (1), which can be achieved by a good policy learning algorithm and well-trained classifiers for reward modification. Secondly, we utilize imitation learning from observation to transfer the experience to the source. (2.1) depends on the upper bound of the importance weight, which can be decreased with a richer policy class or when the dynamics shift becomes smaller. Additionally, a better imitation can be also achieved by increasing the complexity of the discriminator function class and the number of samples, which pushes (2.2) to be smaller.

4.1 Comparison with the Analysis of DARC

As we discussed in Section 3.1, the DARC algorithm [3] trains a policy π DARC on the source domain via matching the distribution of trajectories generated by π DARC in the source and the distribution of the optimal trajectory in the target domain. Consequently, the learned policy π DARC will be suboptimal if it is directly deployed in the target domain.

In the DARC analysis, it is assumed that the optimal policy for the target domain π ∗ lies in the no exploit set defined as follows [3, Assumption 1].

Here, the no exploit set means that the experiences for any policy in this set are similar in the source and target domains. Consequently, any two policies in this no exploit set also receive similar expected rewards in the two domains, and thus the reward received by π ∗ in the target domain is similar to

that received by π DARC in the target domain. Further, the objective function Eq. (3.3) of DARC is equivalent to the following constrained optimization.

Thus, deploying the policy π DARC will not receive a huge performance degradation. However, the assumption that π ∗ ∈ Π no exploit is stringent and might not always be satisfied when the dynamics shift is large. When this assumption is violated, π ∗ is not a good policy in the source domain, though it is the optimal policy in the target domain. Thus, the DARC policy which only optimizes the modified reward in the source domain will have significant performance degradation, as we have empirically shown in Figure 1 (a) and Figure 9. We also demonstrate this performance gap in Lemma A.1 in Appendix A when their assumption is not satisfied.

In contrast, our algorithm DARAIL does not assume the performance of π DARC in the source domain to be close to the performance of π ∗ in the target domain. Instead, we only assume that the importance weight is somehow bounded, meaning that the dynamics shift is bounded. The error bound of our algorithm presented in Theorem 4.1 is controlled by imitation learning, which transfers the performance of π DARC in the source domain to that of π ∗ in the target domain without assuming π ∗ ∈ Π no exploit. Therefore, our algorithm can work well even in the cases shown in Figure 1 (a) and Figure 9 where the experience of π DARC is very distinctive in the source and target domains.

5 Experiment

In this section, we conduct experiments on off-dynamics reinforcement learning settings on four OpenAI environments: HalfCheetah-v2 , Ant-v2 , Walker2d-v2 , and Reacher-v2 . We compare our method with seven baselines and demonstrate the superiority of the proposed DARAIL.

5.1 Experiments Setup

Dynamics Shifts: We examine our algorithm with two types of dynamics shift. 1) Broken environment. Following previous work [3], we freeze the 0 -index value to 0 in action: zero torque is applied to this joint, regardless of the commanded torque. Different from DARC [3], who only test their method in intact source and broken target environment, we further test our algorithm in the broken source and intact target environment, where the source has less support than the target domain. As discussed in Section 4.1, violating the π ∗ ∈ Π no exploit assumption leads to significant performance degradation for DARC and similar methods. When the source domain is intact, this assumption is more likely to hold and DARC can achieve a near-optimal policy in the target domain. So, besides the setting in DARC, we focus on a harder problem for off-dynamics RL where DARC is prone to failure due to the violation of the assumptions in Section 4.1. Further, for the Ant and Walker2d, the source environment is broken with p f = 0 . 8 probability, which means that with 0.8 probability, the 0 -index will be set to be 0, and 0.2 probability remains the original value. More details about the broken environment will be introduced in the Appendix C.3. 2) Modify parameters of the environment. Besides the broken environment, we create dynamics shifts by modifying MuJoCo's configuration files for the target domain. Specifically, we modify one of the coefficients of { gravity , density } from 1.0 to one of the value { 0 . 5 , 1 . 5 } .

Baselines: We first compare our method with DARC performance in the source and target domains. DARC Training and DARC Evaluation , defined as E p src ,π DARC [ ∑ t r ( s t , a t )] and E p trg ,π DARC [ ∑ t r ( s t , a t )] respectively, represent DARC performance in the two domains. We compare DARAIL with DARC training performance as we mimic the DARC behavior in the source domain, which should receive a similar reward as the DARC training reward in the source domain. We compare with DARC Evaluation to show that our method mitigates the problem of DARC and outperforms DARC in the target domain. Further, we compare our method DARAIL with several baselines that we describe as follows. Importance Sampling for Reward ( IS-R ) re-weights the reward in the transition with p trg ( s t +1 | s t ,a t ) p src ( s t +1 | s t ,a t ) , and update the policy with reward p trg ( s t +1 | s t ,a t ) p src ( s t +1 | s t ,a t ) r ( s t , a t ) [18]. Importance Sampling for SAC Actor and Critic Loss ( IS-ACL ) [18] re-weights the transitions in the SAC actor and critic loss. DAIL is a reduction of DARAIL without reward augmentation. Model-based RL method MBPO [19] uses short model rollouts branched from real data to reduce the compounding errors of inaccurate models and decouple the model horizon from the task horizon. MATL [20] uses different modified rewards and is similar to our problem setting, except that they have access to rewards in

the target domain. Finally, we compare with generative adversarial reinforced action transformation ( GARAT ) [10], a grounded action transformation method that uses imitation learning to modify the action that is executed in the source domain to simulate the target transitions. More details of the baselines are in Appendix C.2.

Experimental Details: We perform weight clipping to all methods that use the importance weight, including the DARAIL, DAIL, IS-R, and IS-ACL, and select the [0 . 01 , 100] as the clipping interval for fair comparison, which works well for all methods. We also show that DARAIL is less sensitive to the importance of weight clipping in the next section. We conduct fair parameter tuning for our method and baselines, including learning rate, Gaussian noise scale, and learning frequency of the importance weight. We also tune the parameter for the imitation learning component in DARAIL and DAIL and notice that the higher update frequency tends to perform better, and experiment results are in Appendix D.2. More details are in Appendix D.4.

5.2 Results

We show the results of DARAIL and DARC in Table 1 and 2 for broken source and 1.5 gravity setting, respectively. And the results of other baselines are in Table 3 and 4. We refer to the results on other settings in the Appendix, including the intact source and broken target environment setting and the modification of different scales of the parameters in the configuration file. We will also empirically discuss why DARC works well in the broken target setting while fails in the broken source setting in Appendix C.6.

The Suboptimality of DARC and DARAIL outperforms DARC By comparing DARC Training and DARC Evaluation in Table 1 and 2 we demonstrate that there is a performance degradation of π DARC deployed in the target domain on all four environments. π DARC reward in the target domain is about 40% lower than π DARC reward in the source domain on average for broken source setting, and the degradation can be more severe in the changing gravity and density setting. Also, π DARC reward in the target domain is significantly lower than the target optimal reward. The training reward curves of DARC of the broken source environment setting are in Appendix C.5, clearly showing performance degradation when deployed in the target domain. Further, DARAIL outperforms the DARC evaluation performance.

DARAIL Outperforms Baselines We show the result of DARAIL and baselines in Table 3, 4. The training curves of other settings are in Appendix C.4. In all four environments, DARAIL outperforms the π DARC reward in the target domain. DARAIL also achieves better performance or the same level of rewards compared to the π DARC in the source domain as shown in Table 1 and 2, which is our expert policy for the imitation step. Compared with the DAIL, DARAIL has a much better performance, which demonstrates the effectiveness of the reward estimator R AE . Compared with the two important weighting methods, IS-R and IS-ACL, in broken source settings, DARAIL outperforms IS-R in four environments and IS-ACL in Ant and Walker2d. IS-ACL and DARAIL achieve similar rewards in HalfCheetah and Reacher. And in modifying configuration settings, DARAIL outperforms IS-R and IS-ACL. Our method outperforms MBPO, MATL, and GARAT in all environments.

DARAIL is Less Sensitive to Extreme Values in Importance Weights Though IS-ACL achieves comparable performance with DARAIL on some tasks shown in Table 3, it is highly sensitive to

the clipping interval of importance weight. In Figure 2, we show the performance of DARAIL and IPS-ACL on different importance weight clipping intervals in the broken source setting, and DARAIL outperforms IPS-ACL on all tasks. If the clipping interval is too large, IPS-ACL suffers from high variance, thus harming the performance. If the clipping interval is too small, the effective information about the dynamics shift is lost. On the other hand, DARAIL is less sensitive to it, which is an inherent property of our R AE . Furthermore, in Figure 2, for IPS-ACL, the training curve for [0 . 001 , 1000] clipping interval has a much larger variance than [0 . 1 , 10] clipping interval, while our method does not suffer from such a high variance. This also demonstrates that our proposed reward estimator R AE is a more robust estimator and less affected by the importance weight.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training Steps

1e6

0

1000

2000

3000

4000

5000

6000

7000

Rewards

DARC Training

DARC Evaluation

DARAIL

IS-ACL

0

1

2

3

4

5

6

Training Steps

1e6

0

1000

2000

3000

4000

5000

6000

7000

Rewards

DARC Training

DARC Evaluation

DARAIL

IS-ACL

0

1

2

3

4

Training Steps

1e6

0

500

1000

1500

2000

2500

3000

3500

4000

Rewards

DARC Evaluation

DARC Training

DARAIL

IS-ACL

0

1

2

3

4

Training Steps

1e6

0

1000

2000

3000

4000

Rewards

DARC Evaluation

DARC Training

DARAIL

IS-ACL

(a) HalfCheetah

(b) HalfCheetah

(c) Walker2d

(d) Walker2d

[0

.

1

,

10]

[0

.

001

,

1000]

[0

.

1

,

10]

[0

.

001

,

1000]

DARAIL's Performance on Different Magnitudes of Shifts In our broken action environments, as we create the off-dynamics shift by (probabilistically) freezing one action dimension in the source domain, we can control the off-dynamics shift magnitudes by controlling the broken probability. For the same environment, the larger the p f is, the higher the probability of freezing the 0-index action, thus a larger dynamics shift. We consider p f = [0 . 2 , 0 . 5 , 0 . 8] for Ant, respectively and the experiment results is shown in Figure 3. From left to right, as the dynamics shift increases, we observe that the DARC performance decreases, and DARAIL outperforms DARC on all tasks.

6 Related Work

Off-dynamics RL Off-dynamics RL [3] is a specific domain adaptation [21, 22] and transfer learning problem in the RL domain [23] where the goal is to learn a policy from a source domain to adapt to a target domain where the dynamics are different. Similar to many works in off-policy evaluation (OPE) [12] in bandit and offline/off-policy RL [13, 24], an importance weight approach can be used to account for the difference between the transition dynamics with p trg ( s t +1 | s t ,a t ) p src ( s t +1 | s t ,a t ) . However, this method can easily suffer from high variance due to the estimation bias of p src ( s t +1 | s t , a t ) [12]. Another line of method for the off-dynamics RL is through reward shaping [3, 5]. DARC [3] learns a policy from a modified reward function that accounts for the dynamics shifts through a trajectories distribution matching objective. [6] proposed an unsupervised domain adaptation method with KL regularized objective, which uses the same reward modification techniques trajectories distribution matching

0

1

2

3

4

5

6

Training Steps

1e6

0

1000

2000

3000

4000

5000

6000

Rewards

DARC Evaluation

DARC Training

DARAIL

0

1

2

3

4

5

6

Training Steps

1e6

0

1000

2000

3000

4000

5000

6000

Rewards

DARC Evaluation

DARC Training

DARAIL

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Training Steps

1e6

-1000

0

1000

2000

3000

4000

5000

Rewards

DARC Evaluation

DARC Training

DARAIL

(a) Ant,

p

f

= 0

.

2

(b) Ant,

p

f

= 0

.

5

(c) Ant,

p

f

= 0

.

8

objective in DARC [3]. These reward-shaping methods all face the same problem: they will not recover the optimal policy in the target domain and will suffer from performance degradation in the target domain, but the policy's experience in the source domain is similar to the optimal policy in the target domain. Similarly, [25] proposes a state-regularized policy optimization method that constrains the state distribution to be similar in the source and target domain by adding a constraint term in the reward. However, this will also lead to suboptimal policy in the target domain like DARC. Different from DARC, Mutual Alignment Transfer Learning (MATL) [20] uses different modified rewards with GAN [26] to align the trajectories generated in the source and the target domain, but it requires access to the target domain reward. There is also work [27] that solves the off-dynamics RL problem by training a distributionally robust policy in the source domain by assuming that the target domain's transition probability is in an ambiguity set defined around the transition probability of the source domain. Our method builds on DARC, inspired by its property in the source domain, overcoming the issues in DARC and similar methods by mimicking the π DARC behavior in the source domain.

Imitation Learning Imitation learning (IL) is another line of work that can be applied to off-dynamics problems by mimicking the expert demonstration in the target domain. Generative adversarial imitation learning, [7, 28-30, 8, 31, 32], frames IL as an occupancy-measure matching or divergence minimization problem, which minimizes the divergence of the generated trajectories and the expert demonstration. Building on GAN [26], it uses the RL algorithm as a generator and a classifier as a discriminator to achieve this. Imitation learning from observation ( Ifo ) [33-35] is recently proposed to mimic the expert's behavior without knowing which actions the expert took. In the off-dynamics RL setting, recent work on IL under dynamics mismatch [11, 10, 36] can transfer a policy learned in the source to the target domain with minimal interaction with the target domain. However, these methods require high-quality and sufficient expert demonstrations and also the expert demonstrations might not be the optimal trajectories for the target domain, resulting in a suboptimal policy. Our method, DARAIL, transfers the DARC policy's behavior in the source to the target domain through imitation learning from observation so that the new policy will behave like the optimal policy in the target domain. Furthermore, we propose a novel and practical reward estimator with the signal from the discriminator and the reward from the source domain for the policy optimization.

7 Conclusion

In this paper, we propose Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) for off-dynamics RL. We recognize the drawbacks of DARC and its following work with the same modified rewards function. We demonstrate that DARC or similar reward modification methods can only obtain a near-optimal policy in the target domain. We then propose to mimic the trajectory distribution generated by DARC in the source domain. Specifically, we propose a reward-augmented estimator for the policy optimization step in imitation learning from observation. Theoretically, we established the finite sample upper bounds of rewards for the proposed method, relaxing the restrictive assumption about the optimal policy in the previous work. Empirically, we conducted experiments on four Mujoco environments, demonstrating the superiority of our method. From the safety perspective, our method avoids directly training a policy in a high-risk environment. Our future work includes investigating off-dynamics reinforcement learning under safety constraints and more severe domain gaps in reinforcement learning.

Acknowledgments

We would like to thank the anonymous reviewers for their helpful comments. YG was supported by the Center for Digital Health and Artificial Intelligence (CDHAI) of the Johns Hopkins University. PX was supported in part by the National Science Foundation (DMS-2323112) and the Whitehead Scholars Program at the Duke University School of Medicine. AL was partially supported by the Amazon Research Award, the Discovery Award of the Johns Hopkins University, and a seed grant from the JHU Institute of Assured Autonomy. The views and conclusions in this paper are those of the authors and should not be interpreted as representing any funding agency.

References

Ying Liu, Brent Logan, Ning Liu, Zhiyuan Xu, Jian Tang, and Yangzhi Wang. Deep reinforcement learning for dynamic treatment regimes on medical registry data. In 2017 IEEE international conference on healthcare informatics (ICHI) , pages 380-385. IEEE, 2017.

B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2021.

Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arXiv preprint arXiv:2006.13916 , 2020.

Junda Wu, Zhihui Xie, Tong Yu, Qizhi Li, and Shuai Li. Sim-to-real interactive recommendation via off-dynamics reinforcement learning. In 2rd Offline Reinforcement Learning Workshop Advances at NeurIPS , 2021.

Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. arXiv preprint arXiv:2203.06662 , 2022.

Jinxin Liu, Hao Shen, Donglin Wang, Yachen Kang, and Qiangxing Tian. Unsupervised domain adaptation with dynamics-aware rewards in reinforcement learning. Advances in Neural Information Processing Systems , 34:28784-28797, 2021.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158 , 2018.

Shengyi Jiang, Jingcheng Pang, and Yang Yu. Offline imitation learning with a misspecified simulator. Advances in neural information processing systems , 33:8510-8520, 2020.

Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems , 33:3917-3929, 2020.

Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. arXiv preprint arXiv:2002.11879 , 2020.

Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv:1103.4601 , 2011.

Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning , pages 652-661. PMLR, 2016.

Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning , pages 9167-9176. PMLR, 2020.

Tengyu Xu, Zhuoran Yang, Zhaoran Wang, and Yingbin Liang. Doubly robust off-policy actor-critic: Convergence and optimality. In International Conference on Machine Learning , pages 11581-11591. PMLR, 2021.

Nathan Kallus, Xiaojie Mao, Kaiwen Wang, and Zhengyuan Zhou. Doubly robust distributionally robust off-policy evaluation and learning. In International Conference on Machine Learning , pages 10598-10632. PMLR, 2022.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Joshua Arvind Holla. On the off-dynamics approach to reinforcement learning . McGill University (Canada), 2021.

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems , 32, 2019.

Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. In Conference on Robot Learning , pages 281-290. PMLR, 2017.

Thomas Carr, Maria Chli, and George Vogiatzis. Domain adaptation for reinforcement learning on the atari. arXiv preprint arXiv:1812.07452 , 2018.

Jinwei Xing, Takashi Nagata, Kexin Chen, Xinyun Zou, Emre Neftci, and Jeffrey L Krichmar. Domain adaptation in reinforcement learning via latent unified state representation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10452-10459, 2021.

Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Zhenghai Xue, Qingpeng Cai, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. State regularized policy optimization on data with dynamics shift. Advances in neural information processing systems , 36, 2024.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Zhishuai Liu and Pan Xu. Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation. In International Conference on Artificial Intelligence and Statistics , pages 2719-2727. PMLR, 2024.

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248 , 2017.

Kee-Eung Kim and Hyun Soo Park. Imitation learning via kernel mean embedding. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. arXiv preprint arXiv:1810.00821 , 2018.

Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, and Huaping Liu. Task transfer by preference-based cost learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 2471-2478, 2019.

Faraz Torabi, Garrett Warnell, and Peter Stone. Imitation learning from video by leveraging proprioception. arXiv preprint arXiv:1905.09335 , 2019.

YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1118-1125. IEEE, 2018.

Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954 , 2018.

Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observation. arXiv preprint arXiv:1905.13566 , 2019.

Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning , pages 5286-5295. PMLR, 2020.

Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International conference on machine learning , pages 224-232. PMLR, 2017.

Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems , 33:15737-15749, 2020.

A Analysis of DARC

A.1 DARC Objective

Figure 4 shows the objective of DARC, which minimizes the reverse KL divergence of the trajectories generated by the π DARC in the source domain and π ∗ in the target domain. Note that the optimal policy is assumed to be proportional to the exponential form of the reward, i.e. π ∗ ∝ exp( r ( s t , a t )) . Given this assumption, the reverse KL divergence can be re-formulated to Eq. (3.3) with modified reward. So, the π DARC will not be optimal in the target domain but can generate trajectories in the source domain that resemble the optimal trajectories given the objective.

<latexi sh

1_b

64="vf+MOWCX

kD7

B/Zj9

qm3

yT0

>A

H

c

V

NS8

EJ

Ur

I

Q

o

Y

g

Gz

L

5u

w

Pp

K

nd2

R

F

<latexi sh

1_b

64="

YKwH

gqB

9Ic0

fr+SROvA

>

7

V

N

8

EJ3

W

X

5

FL

m

Q

ZT

u

Dd

C

/

2z

p

y

P

o

n

G

k

U

j

M

<latexi sh

1_b

64="vf+MOWCX

kD7

B/Zj9

qm3

yT0

>A

H

c

V

NS8

EJ

Ur

I

Q

o

Y

g

Gz

L

5u

w

Pp

K

nd2

R

F

<latexi sh

1_b

64="UmG

H2

8Mn5

d

o

>

B

c

V

NS

EJ

7

T

R

9k

q

r

v3

Z

u

yQp

/

0

P

g

D

z

K

O

f

W

L

F

I

j

Target Domain

Minimize Reverse KL Divergence

Source Domain

(

s

t

, s

t

+1

)

(

s

t

, s

t

+1

)

π

DARC

π

∗

A.2 DARC Error Bound

Now, we show that without the assumption of π ∗ ∈ Π no exploit in [3], the error of π DARC cannot be trivially bounded.

Lemma A.1. If π ∗ / ∈ Π no exploit, the error bound of the π DARC is in the following form:

Proof. In [3] Lemma B.2, they show that for any policy π ∈ Π no exploit , the following inequality holds:

where R max refers to the maximum entropy-regularized return of any trajectories. However, the inequality Eq. (A.1) only holds for π DARC, not for π ∗ . Now, we show that without the assumption π ∗ ∈ Π no exploit , the error could not be bounded trivially.

We start with the same decomposition. Therefore, we have

In the original proof of [3], they bound the three terms based on the following idea:

For the term I 1 , they directly assume π ∗ ∈ Π no exploit and obtain I 1 ≤ 2 R max √ ϵ/ 2 based on inequality Eq. (A.1). However, without the π ∗ ∈ Π no exploit , the upper bound is not valid. A valid upper bound should be:

If π ∗ ∈ Π no exploit holds, we have D KL ( p trg ,π ∗ ( τ ) , p src ,π ∗ ( τ )) ≤ ϵ , which recovers the inequality Eq. (A.1). If it doesn't, we cannot trivially bound the D KL ( p trg ,π ∗ ( τ ) , p src ,π ∗ ( τ )) .

For the term I 2 , in the proof of [3], they also assume π ∗ ∈ Π no exploit and obtain the I 2 ≤ 0 based on the objective π DARC maximizes the reward in the source domain with π DARC ∈ Π no exploit . If π ∗ ∈ Π no exploit doesn't hold, we can bound this term by the following inequality:

which is equivalent to

And the total variation of the two policies cannot be trivially bound as well. For the term I 3 , we can easily bound it by applying the inequality Eq. (A.1) as π DARC ∈ Π no exploit .

In summary, the bound without assuming π ∗ ∈ Π no exploit will be:

This completes the proof.

B Theoretical Analysis of DARAIL

In this section, we prove our theoretical results.

Definition B.1. (Neural Network Distance [37, 38]) For a class of neural networks D , the neural network distance between two state-next state distributions, τ src π DARC and τ trg ζ , is defined as

Definition B.2. (Empirical Rademacher Complexity) Given a function class F , a dataset X = ( x 1 , x 2 , ..., x n ) , i.i.d drawn from distribution µ and random variable σ defined as P ( σ = 1) = P ( σ = -1) = 1 2 , the empirical Rademacher complexity is given by:

Definition B.3. (Linear Span of the Discriminator) Consider a span of the discriminator class: span( D ) = { c 0 + ∑ k i c i D i : c 0 ∈ R , D i ∈ D , n ∈ N } . Assuming the ground truth reward function r lies in the span( D ), then the compatible coefficient is defined as:

The compatible coefficient represents the minimum number of functions in D required to the reward function r , which means the complexity of the reward function r .

Lemma B.4. (GAIL Generalization). Let π DARC be the expert policy and ˆ ζ be the solution of the imitation learning algorithm. Let discriminator class D be a ∆ -bounded function, i.e. | D ( s t , s t +1 ) | ≤ ∆ . Suppose reward function r lies in the span of the discriminator class. Given d D (ˆ τ src π DARC , ˆ τ trg ζ ) -inf ζ d D (ˆ τ src π DARC , ˆ τ trg ζ ) ≤ ˆ ϵ (empirical neural network distance achieved by imitation learning), the importance weight ρ ( s, s t +1 ) is bounded by W , m is the number of the expert data, then ∀ δ ∈ (0 , 1) , with probability at least 1 -δ , we have

Proof. Given d D (ˆ τ src π DARC , ˆ τ trg ζ ) -inf ζ d D (ˆ τ π src DARC , ˆ τ trg ζ ) ≤ ˆ ϵ , we can have

We prove that d D ( τ src π DARC , τ trg ζ ) -d D (ˆ τ src π DARC , ˆ τ trg ζ ) has an upper bound.

According to McDiarmid 's inequality, with probability at least 1 -δ 2 , the following inequality holds

By a similar derivation, we can have

Thus, we have

Then, based on Theorem 2 in [38], we can conclude that

This completes the proof.

Theorem B.5. Let π ∗ = argmax π E π,p trg [ ∑ t r ( s t , a t )] be the policy maximizing the cumulative reward in the target domain and ˆ ζ be the policy learned from DARAIL. Let m be the number of the expert demonstration and ˆ R ( m ) π = E σ [ sup D ∈D 1 m ∑ m i =1 σ i D ( s t , s t +1 ) ] be the empirical Rademacher complexity. Let B be the error bound of DARC in the source domain, i.e. E p src ,π ∗ DARC [ ∑ t r ( s t , a t ) + H [ a t | s t ]] -E p src ,π DARC [ ∑ t r ( s t , a t )] ≤ B and W be the upper bound

of the importance weight, i.e. ρ ( s t , s t +1 ) ≤ W , ∀ ( s t , s t +1 ) . Let discriminator class D be a ∆ -bounded function, i.e. | D ω ( s t , s t +1 ) | ≤ ∆ given any ( s t , s t +1 ) . ∥ r ∥ D measures the richness of the discriminator to represent the ground truth reward as defined in Appendix B.2. d D is a defined neural network distance between the ( s t , s t +1 ) distributions generated by the π DARC and π ˆ ζ defined in Appendix B.1. Given the empirical training error of the imitation learning, i.e. d D (ˆ τ src π DARC , ˆ τ trg ˆ ζ ) -inf ζ d D (ˆ τ src π DARC , ˆ τ trg ζ ) ≤ ˆ ϵ , ∀ δ ∈ (0 , 1) , with probability at least 1 -δ , we have

︸

Proof. We can first decompose it into three terms:

Based on the formulation, π ∗ DARC can generate optimal trajectories for the target domain in the source domain so that I 1 = 0 . Also, the I 2 term is the training error of the DARC and the entropy term of the optimal DARC policy, and we can assume together they are bounded by B . Then, we only need to bound the I 3 terms. Combining Lemma B.4, we have

︸

C Additional Experimental Details and Results

Code is available at https://github.com/guoyihonggyh/Off-Dynamics-Reinforcement-Learning-viaDomain-Adaptation-and-Reward-Augmented-Imitation.

Following the DARC [3], the importance weight can be estimated with the following two binary classifiers p ( trg | s t , a t ) and p ( trg | s t , a t , s t +1 ) with Bayes' rules:

Replacing the p ( s t , a t | trg ) in Eq. (C.1) with Eq. (C.2), we obtain:

Similarly, we can obtain the p src ( s t +1 | s t , a t ) = p ( src | s t ,a t ,s t +1 ) p ( s t ,a t ,s t +1 ) p ( src | s t ,a t ) p ( s t ,a t ) .

We can calculate the ∆ r ( s t , a t , s t +1 ) following:

ρ ( s t , s t +1 ) can be obtained from ρ ( s t , s t +1 ) = exp [∆ r ( s t , a t , s t +1 )]

Training the classifier p ( trg | s t , a t ) and p ( trg | s t , a t , s t +1 ) The two classifiers are parameterized bu θ SA and θ SAS. To update the two classifiers, we sample one mini-batch of data from the source replay buffer D ζ src and the target replay buffer D ζ src respectively. Imbalanced data is considered here as each time we sample the same amount of data from the source and target domain buffer. Then, the parameters are learned by minimizing the standard cross-entropy loss:

Thus, θ = ( θ SAS , θ SA ) is obtained from:

C.2 Description of Baseline Methods

Importance Sampling for Reward (IS-R) With ( s t , a t , s t +1 ) from the source domain, the IS-R directly re-weight the reward in each transition. We can view IS-R as learning the SAC with rewards p trg ( s t +1 | s t ,a t ) p src ( s t +1 | s t ,a t ) r t ( s t , a t ) and seeking to maximize the following objective:

Importance Sampling for SAC Actor and Critic Loss (IS-ACL) Another way of doing importance sampling is by re-weighting the actor and critic loss in SAC. The loss for the Q-network in SAC becomes:

where d is the done signal, and the target is given by:

The actor loss is:

DAIL In DARAIL, the policy is optimized with the reward estimator R AE with the true reward from the source domain. We also want to compare the vanilla imitation learning with importance weight. The objective is:

Then, following the Eq.(C.3), the objective of policy optimization without the reward estimator is:

We can view it as a reduced version of our proposed method, which uses the reward function provided by the discriminator and importance weight.

MBPO [19]. MBPO is a model-based RL method. We train the MBPO in the source domain and deploy it to the target domain.

MATL [20]. MATL modified the reward on both the source and target domains and aligned the trajectories on both domains. Unlike our method, they need access to the reward from the target domain.

GARAT [10] GARAT is a grounded action transformation approach that simulates target transitions ( s t , a t , s t +1 , r ) in the source domain with modified action, where the modified action is learned from imitation learning.

C.3 Broken with probability p f

As discussed, we use the broken with probability for Ant and Walker2d. The dynamics shift created by freezing one action varies across environments. For instance, in the Ant robot, the 0 -index controls the rotor between the torso and front left hip, while in the HalfCheetah, the 0 -index controls the back thigh rotor. So, the broken Ant experiences a larger shift than the broken HalfCheetah if we break the 0 -index for both environments. Also, the broken environment in Walker2d and Ant creates such a large dynamics shift that it is overly difficult to adapt from the source domain, i.e., DARC cannot obtain the optimal reward in the source domain. We then introduce the broken with probability p f to better control the magnitude of dynamics shift. Broken with probability p f means the 0 -index action is frozen with probability p f and follows the commanded torque with probability 1 -p f . In Reacher and HalfCheetah, the source environment is broken with probability 1 . Ant and Walker2d's source domain is broken with a probability of 0 . 8 .

Figure 5 shows the performance of DARC in Ant and Walker2d under different broken probability p f in the source domain. We can observe that when p f = 1 . 0 , the performance degradation of evaluating in the target domain is larger than the p f = 0 . 8 case. Also, when p f = 1 . 0 , the DARC evaluation performance in the target domain is close to 0. Moreover, we notice that in the p f = 1 . 0 case, DARC training performance in the source domain receives a much lower reward than the p f = 0 . 8 case. However, we want to mimic the DARC behavior in the imitation learning, so we want DARC to be able to receive optimal reward in the source domain. Thus, for the Ant and Walker2d environment, we choose p f = 0 . 8 for the source domain.

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

Reward

Training Reward

Evaluation Reward

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

Reward

Training Reward

Evaluation Reward

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

1000

2000

3000

4000

Training Reward

Evaluation Reward

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

1000

2000

3000

4000

Reward

Training Reward

Evaluation Reward

Optimal Reward in Target

(a) Ant,

p

f

= 0

.

8

(b) Ant,

p

f

= 1

(c) Walker2d,

p

f

= 0

.

8

(d) Walker2d,

p

f

= 1

C.4 Training Curve of the DARAIL and Baselines

We show the training curve of DARAIL and baselines in different environments under the broken source environment setting in Figure 6 corresponding to the result in Table 3. We also show the training curve of modifying the configuration in Figure 7 and 8.

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Training Steps

1e6

0

2000

4000

6000

8000

HalfCheetah

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Training Steps

1e6

-1000

0

1000

2000

3000

4000

5000

6000

Ant

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Training Steps

1e6

0

1000

2000

3000

4000

5000

Walker2d

0

10000

20000

30000

40000

Training Steps

-70

-60

-50

-40

-30

-20

-10

Reacher

DARC Evaluation

DARC Training

DARAIL

IS-R

IS-ACL

MBPO

MATL

GARAT

Optimal Reward in Target

DARC Evaluation

DARC Training

DARAIL

IS-R

IS-ACL

MBPO

MATL

GARAT

Optimal Reward in Target

Figure 6: Upper horizon line: DARC reward in the source domain. Lower horizon line: DARC reward in the target domain. The figures show the mean value of multiple runs and the standard deviation. The figure shows that our proposed method performs better than DARC in the target domain and other baseline methods.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training Steps

1e6

0

2000

4000

6000

8000

HalfCheetah

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Training Steps

1e6

-1000

0

1000

2000

3000

Ant

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Training Steps

1e6

0

500

1000

1500

2000

2500

Walker2d

0

20000

40000

60000

80000

Training Steps

-40

-30

-20

-10

0

Reacher

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Training Steps

1e6

-1000

0

1000

2000

3000

4000

5000

6000

7000

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Training Steps

1e6

-1000

0

1000

2000

3000

4000

5000

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Training Steps

1e6

0

1000

2000

3000

4000

5000

0

20000

40000

60000

80000

Training Steps

-80

-60

-40

-20

0

DARC Evaluation

DARC Training

DARAIL

IS-R

IS-ACL

MBPO

MATL

GARAT

Optimal Reward in Target

DARC Evaluation

DARC Training

DARAIL

IS-R

IS-ACL

MBPO

MATL

GARAT

Optimal Reward in Target

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Training Steps

1e6

0

2000

4000

6000

8000

10000

HalfCheetah

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Training Steps

1e6

-2000

-1000

0

1000

2000

3000

Ant

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Training Steps

1e6

0

500

1000

1500

2000

2500

Walker2d

0

20000

40000

60000

80000

Training Steps

-70

-60

-50

-40

-30

-20

-10

0

Reacher

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Training Steps

1e6

-2000

0

2000

4000

6000

8000

10000

12000

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training Steps

1e6

-2000

0

2000

4000

6000

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training Steps

1e6

0

500

1000

1500

2000

2500

3000

0

20000

40000

60000

80000

Training Steps

-70

-60

-50

-40

-30

-20

-10

0

DARC Evaluation

DARC Training

DARAIL

IS-R

IS-ACL

MBPO

MATL

GARAT

Optimal Reward in Target

DARC Evaluation

DARC Training

DARAIL

IS-R

IS-ACL

MBPO

MATL

GARAT

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

8000

Reward

Training Reward

Evaluation Reward

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

Reward

Training Reward

Evaluation Reward

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

1000

2000

3000

4000

Training Reward

Evaluation Reward

Optimal Reward in Target

0

20000

40000

Training Steps

-40

-30

-20

-10

Reward

Training Reward

Evaluation Reward

Optimal Reward in Target

(a) HalfCheetah

(b) Ant

(c) Walker2d

(d) Reacher

0.0

0.5

1.0

1.5

Training Steps

1e6

0

2000

4000

6000

8000

10000

12000

DARAIL

DARC Evaluation

DARC Training

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

-2000

0

2000

4000

DARAIL

DARC Evaluation

DARC Training

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

500

1000

1500

DARAIL

DARC Evaluation

DARC Training

Optimal Reward in Target

0

20000

40000

60000

80000

Training Steps

-35

-30

-25

-20

-15

-10

-5

DARAIL

DARC Evaluation

DARC Training

Optimal Reward in Target

(a) HalfCheetah

(b) Ant

(c) Walker2d

(d) Reacher

C.5 DARC training and evaluation performance on broken source setting

Figure 9 shows the performance of DARC trained in the source and evaluated in the target domain under broken source environment setting. The training reward is the reward obtained in the source domain, i.e. E π DARC ,p src [ ∑ t r ( s t , a t )] and the evaluation is the reward deployed in the target domain, i.e. E π DARC ,p trg [ ∑ t r ( s t , a t )] . We observe the performance degradation in the figure 9. Empirically, we notice that the DARC policy performance in the source domain, E π DARC ,p src [ ∑ t r ( s t , a t )] , is close to the optimal reward in the target domain which matches with the DARC objective that DARC can generate target optimal trajectories in the source domain. However, deploying it to the target domain will result in performance degradation and a suboptimal reward due to the dynamics shift.

C.6 Performance of DARAIL on broken target environment

We show the performance of DARAIL in the intact source and broken target environment setting in Figure 10 (the setting in DARC paper [3]). We observe that our method outperforms the DARC reward in the target domain, E π DARC ,p trg [ ∑ t r ( s t , a t )] . Also, we see that the performance of DARC in the source domain and target domain are very similar. Compared with the performance gap when the source environment is broken in Figure 9. As discussed, DARC works well when the assumption that the target optimal policy performs well in the source domain is satisfied. In the broken target setting, the target optimal policy can perform the same in the source domain.

Further, empirically, in the broken target setting, the DARC policy learns a near 0 value for the broken joint, which guarantees that the policy can generate similar trajectories in the two domains. Also, maximizing the adjusted cumulative reward in the source domain with a policy with a near 0 value for the broken joint is equivalent to maximizing the cumulative reward in the target domain. Thus, DARC perfectly suits the broken target setting. However, in the broken source setting and other more general dynamics shift cases, the target optimal policy might not perform well in the source domain. For example, in the broken source setting, the target optimal policy will perform poorly in the source domain as it loses one joint in the source domain. Another way to understand why DARC fails is that it learns an arbitrary value for the broken joint, which becomes detrimental in the target domain. However, this is just an artifact of the particular setting. As we discussed above, the intrinsic reason that DARC fails is the violation of the assumption.

C.7 Performance of mimicking source optimal trajectories

In Figure 11, We compare our DARAIL, which uses DARC trajectories in the source domain as expert demonstrations and mimicking source optimal trajectories regardless of the target domain.

C.8 Access to the target domain data compared to DARC.

Both DARC and DARAIL require some limited access to the target rollouts. In DARAIL, the imitation learning step only rolls out data from the target domain every 100 steps of the source domain rollouts, which is 1% of the source domain rollouts. We claim that more target domain rollouts will not improve DARC's performance due to its suboptimality, and DARAIL is better not because of having more target domain rollouts. We verify it by comparing DARC and DARAIL with the same amount of rollouts from the target domain in the broken source environment setting

0.0

0.5

1.0

1.5

Training Steps

1e6

0

2000

4000

6000

8000

DARAIL

Optimal Reward in Source

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

DARAIL

Optimal Reward in Source

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

-500

0

500

1000

1500

2000

2500

3000

DARAIL

Optimal Reward in Source

Optimal Reward in Target

0

20000

40000

60000

80000

Training Steps

-40

-30

-20

-10

0

DARAIL

Optimal Reward in Source

Optimal Reward in Target

0.0

0.5

1.0

1.5

2.0

2.5

Training Steps

1e6

-2000

0

2000

4000

6000

8000

DARAIL

Optimal Reward in Source

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

-1000

0

1000

2000

3000

4000

5000

6000

DARAIL

Optimal Reward in Source

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

500

1000

1500

2000

DARAIL

Optimal Reward in Source

Optimal Reward in Target

0

20000 40000 60000 80000100000 120000

Training Steps

-40

-30

-20

-10

0

DARAIL

Optimal Reward in Source

Optimal Reward in Target

(a) HalfCheetah

(b) Ant

(c) Walker2d

(d) Reacher

in Tables 11 and 12. Specifically, we examine DARAIL with 5e4 target rollouts alongside DARC with 2e4 and 5e4 target rollouts. DARAIL has 5e3 target rollouts for the Reacher environment, while DARC has 3e3 and 5e3 rollouts. From the results, we see that increasing the target rollouts from 2e4 to 5e4 (or from 3e3 to 5e3 in the case of Reacher) does not yield a significant improvement in DARC's performance due to its inherent suboptimality. Notably, DARAIL consistently outperforms DARC when given comparable levels of target rollouts.

0.0

0.5

1.0

1.5

2.0

2.5

Training Steps

1e6

-2000

0

2000

4000

6000

Rewards

HalfCheetah

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Training Steps

1e6

-1000

0

1000

2000

3000

4000

5000

Rewards

Walker2d

DARC Evaluation

DARC Training

n = 1

n = 10

n = 50

DARC Evaluation

DARC Training

n = 1

n = 10

n = 50

0.0

0.5

1.0

1.5

2.0

2.5

Training Steps

1e6

-1000

0

1000

2000

3000

4000

5000

6000

7000

Rewards

HalfCheetah

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Training Steps

1e6

-1000

0

1000

2000

3000

4000

Rewards

Walker2d

DARC Evaluation

DARC Training

n = 1

n = 10

n = 50

DARC Evaluation

DARC Training

n = 1

n = 10

n = 50

D Ablation Study

D.1 Per-Step Importance Weight v.s Cumulative Importance weight

In our paper, to reduce the variance, we use the per-step importance weight p trg ( s t ,s t +1 ) p src ( s t ,s t +1 ) for the importance sampling method and DARAIL. Here, we compare the per-step importance weight with the cumulative n-step importance weight, which is the multiplication of the weight before time step t :

Note that here, the importance weight is the multiplication of the last n steps weight instead of the multiplication from i = 0 to i = t . Because the cumulative importance weight might have a NaN value due to the product. Thus, the optimization step for the imitation learning of DARAIL is as follows:

Similarly, the objective of IS-R is:

Wecompare the per-step importance weight and the cumulative n-step importance weight on DARAIL and IS-R. Specifically, we consider n = [10 , 50] for HalfCheetah and Walker2d, respectively. We show the results of DARAIL in Figure 12 and the results of IS-R in Figure 13. We see that the cumulative importance weight doesn't perform well on both methods and environments. In HalfCheetah, we can observe that the 10-step cumulative importance weight performs better than the 50-step one. And similar patterns appear in the Walker2d. Thus, we can conclude that per-step importance weight will have a lower variance and be more favorable in our experiment.

D.2 Update Steps of Discriminator

0.0

0.5

1.0

1.5

2.0

2.5

Training Steps

1e6

0

1000

2000

3000

4000

5000

6000

7000

Rewards

DARC Evaluation

DARC Training

update steps 10

update steps 50

update steps 500

update steps 1000

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Training Steps

1e6

0

1000

2000

3000

4000

5000

Rewards

DARC Evaluation

DARC Training

update steps 10

update steps 50

update steps 500

update steps 1000

0

10000

20000

30000

40000

Training Steps

-70

-60

-50

-40

-30

-20

-10

Rewards

DARC Evaluation

DARC Training

update steps 50

update steps 100

update steps 1000

update steps 2000

(a) HalfCheetah

(b) Walker2d

(c) Reacher

In imitation learning, we alternatively update the generator and discriminator. In practice, we normally update the generator several steps and then update the discriminator once. The update steps, updating the discriminator every how many training steps, is a hyperparameter and is important in GAN training. The smaller the update steps are, the higher the update frequencies are. We tune the update steps and show the result of it in different environments. The best discriminator update step in HalfCheetah, Walker2d, and Reacher are 50 , 50 , and 1000 , respectively. We varied the discriminator update steps in HalfCheetah and Walker2d in [ 10 , 50 , 500 , 1000 ] steps, and the update steps in Reacher are [50 , 100 , 1000 , 2000] steps. Figure 14 shows the effects of different discriminator update steps in the final performance. As we can see, for all three environments, a smaller update step (higher update frequency) is preferred as it can learn a better reward estimation. However, as we noticed, for example, for HalfCheetah and Walker2d, when the update step is 50, decreasing it to 10 will not further improve the performance.

D.3 Increase the weight on the modified reward of DARC.

We tested DARC algorithm with modified reward r ( s t , a t ) + η ∆( s t , a t , s t +1 ) with η > 1 instead of η = 1 . And the η = 1 is derived from the distribution matching objective in Eq.(3.3). We show the results in Figure 15 under the broken source environment setting. We can see that increasing η will not increase the DARC performance in the target domain but will hurt the performance of DARC in the target domain.

D.4 Hyperparameters

For a fair comparison, we tune the parameters of baselines and our method. The hidden layers of the policy and value network are [256,256] for the HalfCheetah, Ant, and Walker2d and [64,64] for Reacher. And the hidden layer of the two classifiers is [64] for the HalfCheetah, Ant, and Walker2d and [32] for Reacher. The batch size is set to be 256. We regularize the state by adding the running average of the state. We fairly tune the learning rate from [3 e -4 , 1 e -4 , 5 e -5 , 1 e -5] . For those methods that require the importance weight ρ , we tune the update steps of the two classifiers trained to obtain the importance weight from [10 , 50 , 100] . We also add Gaussian noise ϵ ∼ N (0 , 1) to the

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

8000

Reward

Training Reward

Evaluation Reward

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

Reward

Training Reward

Evaluation Reward

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

1000

2000

3000

4000

Training Reward

Evaluation Reward

Optimal Reward in Target

0

20000

40000

Training Steps

-40

-30

-20

-10

Reward

Training Reward

Evaluation Reward

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

8000

Reward

DARC Training

DARC Evaluation

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

8000

Reward

DARC Training

DARC Evaluation

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

1000

2000

3000

Reward

DARC Training

DARC Evaluation

Optimal Reward in Target

0

20000

40000

Training Steps

-80

-60

-40

-20

Reward

DARC Training

DARC Evaluation

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

8000

Reward

DARC Training

DARC Evaluation

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

2000

4000

6000

8000

Reward

DARC Training

DARC Evaluation

Optimal Reward in Target

0

1

2

3

4

Training Steps

1e6

0

1000

2000

3000

Reward

DARC Training

DARC Evaluation

Optimal Reward in Target

0

20000

40000

Training Steps

-60

-40

-20

Reward

DARC Training

DARC Evaluation

Optimal Reward in Target

(a) HalfCheetah

(b) Ant

(c) Walker2d

(d) Reacher

input of the classifiers for regularization, and the noise scale is selected from [0 . 1 , 0 . 2 , 1 . 0] . For the imitation learning component, the number of expert trajectories is 20. We further tune the update steps of the discriminator and add Gaussian noise to the input of the discriminator.

D.5 Computation Resources

We run the experiment on a single GPU: NVIDIA RTX A5000-24564MiB with 8-CPUs: AMD Ryzen Threadripper 3960X 24-Core. Each experiment requires 12GB RAM and require 20GB available disk space for storage of the data.

E Limitations

A potential limitation will be that we rely on DARC or similar methods to generate state pairs. An overly large dynamics shift, or data limitation may prevent us from obtaining high-quality state space data to imitate in the source domain. We do the experiment on the Mujoco environment instead of the real-world sim-2-real problem. We leave the investigation of this to future work.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract and Introduction section states the contribution. And in the introduction section, we have a contribution list.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We talk about the limitation of our method in the Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes. We present our theoretical result in Section 4 and the proof is in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide details about how we create the dynamics shift and the hyperparameters that we used in the experiments in Appendix D.4 and release the code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a GitHub repository in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the details of the experiment setting, including how to create the dynamics shift in the Experiment section. We also describe the hyperparameter tuning in the Appendix D.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have multiple runs of each experiment and report the mean value and standard deviation in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the GPU/CPU as well as the RAM and storage information for each experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more computing than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our data is open source benchmarks in the RL research field.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the conclusion, we briefly mentioned that our method avoids directly training a policy in a high-risk environment in safety-critical tasks.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We run the experiment on the simulated RL benchmarks; thus, no such issue exists.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide citations to all the data and related work in our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We include the code in our paper. Also, details about the implementation are included in the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our experiments are conducted on the RL benchmarks and thus do not involve any crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our research and experiment don't require IRB as we conducted experiments on simulated RL benchmarks.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces an improvement over the previous method (DARC) with theoretical and experimental contributions for solving the off-dynamics transfer learning problem.\n\nSTRENGTHS:\n- Using IL on states is a sensible idea to add\n- The theoretical contribution is quite strong, as the authors were able to replace a very limiting assumption with a possibly more relaxed (and simpler to analyze) assumption\n- There indeed appears to be a performance boost in the environments / transfer schemes tested.\n\nWEAKNESSES:\n- It's not clear how many samples are used as rollout in the target domain. If it is a non-trivial amount, then a baseline should be to run RL in the target domain with this sample budget.\n- Similarly, DARC should also be allowed to have these additional samples in the target domain for proper comparison.\n- Finally, the presentation quality poses a bit of a weakness. It is a bit difficult to read/analyze at times (see next section for suggestions to improve)\n- Figure 3 does not appear to be a significant performance boost, can you comment on the purported gain?\n\nQUESTIONS:\nQuestions about the paper:\nIn the appendix you show that lower discriminator update steps are better, stopping at 50. Can you explain why lower might be better; and why not decrease even further?\n- You state a few times that you do not collect reward data from the target domain. Isn't this trivial since the reward function is identical to that of the source?\n\n\nVarious (some minor) presentation issues and references:\n- suggest to abbreviate as \"GAIL-O\" or similar. Readers will likely know GAIL and thus understand the connection more quickly\n- In main contributions, the first bullet point is quite unclear; can you re-word/shorten? Similarly, I believe you can remove all but the last sentence in bullet point 2.\n- \"works\" -> \"work\" throughout\n- Use of discounting: You use SAC with a discount factor, but $\\gamma$ is missing from the main text and appendix in many equations. Can you include / comment on this? It appears the theory is only relevant for $\\gamma=1$ which is okay but should be stated upfront.\n- Relatedly, it may be beneficial to cite in Sect. 3.1 e.g. https://arxiv.org/abs/1805.00909 ;  https://arxiv.org/pdf/2005.01643 \n- Regarding the dynamics-shift problem, can you also comment on the relationship to e.g. https://proceedings.neurips.cc/paper_files/paper/2023/file/67dd6a41bf9539cffc0fc0165e4d0616-Paper-Conference.pdf ; https://ojs.aaai.org/index.php/AAAI/article/view/25817 ; https://proceedings.mlr.press/v155/pertsch21a/pertsch21a.pdf \n- L80 \"number of data\" -> \"amount of data\"\n- L84: \"to mimic **an** expert\"\n- (nitpicking) L90: discussion of observations vs. states does not sound quite right; I believe the true distinction is in the POMDP framework\n- Eq. 3.1 remove the extra parentheses around product.\n- Eq. 3.3 Do you still need a min on the RHS? (I might be missing something, please explain if so!)\n- the value of $c$ may be nice to see explicitly so that we can see independence on the dynamics and policy.\n- L128: \"Domian\" -> \"Domain\"\n- L135 \"while we do not have much access to the target domain.\" can you be more specific?\n- L150: \"followed\" -> \"follows\"\n- L172: \"Baye's\" -> \"Bayes'\"\n- Algorithm 1; maybe highlight which parts of the algorithm are distinct from DARC?\n- Theorem 4.1: I believe it would be helpful to either (a) give a more formal version of the theorem with all definitions (define H, D_omega, is $\\gamma$ missing?) or (b) make it an \"informal\" theorem, with the full result in the appendix.\n- Which version of the environments are you using?\n- L304: \"DARAIL's ??? on Different...\" is there a missing word (Effect)?\n- Figures: Can you please include a shaded region for the DARC eval/training reward lines? \n- Figure 3: DARRIL- -> DARAIL?\n\n\n\n----\nOverall, I believe the paper needs a significant clean-up of writing and additional experiments and discussion to merit a higher score.\n\nLIMITATIONS:\nAs stated above, two important limitations the authors can further comment on are (a) use of samples in the target domain and (b) gain in performance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides solution to a specific kind of domain adaptation, where the source action space is a subset of the target action space. In this setup, the source policy might fail in the target domain, because the policy would output arbitrary values for the action dimensions only active in the target domain, but not in source. To resolve this, the paper disentangles the effect of actions, and considers the source domain's policy's state trajectories as the optimal trajectories to imitate in the target domain. By applying imitation learning from observation, they can learn a target policy to reproduce the state transitions from the source policy, but using the action space from the target domain. One trivial solution to this is that the target policy would learn the source policy's actions, just with a 0 output (or a value that leads to no effect) for the extra dimensions in the target domain. The paper outperforms the prior work DARC on its specialized environment setup, but unlikely to be a general method for other off-dynamics setups.\n\nSTRENGTHS:\n- The paper provides a good overview of the prior work DARC, and helped to understand their method without having to go through DARC.\n- The experimental evaluation consists of several baselines, and DARAIL generally outperforms them, achieving similar to source performance. This shows that the imitation learning from observation pipeline is working.\n- They provide code for reproducibility.\n\nWEAKNESSES:\n## Evaluation on a very specific kind of dynamics change\nThe major issue with this work is its limited evaluation where the only kind of off-dynamics shift is freezing the 0-index value to 0 in the action in the source domain. Ideally, if the action support of source is a subset of the target domain, then the same policy should be perfectly fine in the target domain. Yet, this paper shows that the same policy underperforms in the target domain. The only reason why this might be happening is because the DARC policy learns an arbitrary value for the 0-th dimension of action torque, which is inconsequential in the source domain, however, becomes detrimental in the target domain. The paper never discusses this obvious reason for this failure, and the only thing a method needs to do is to learn to always output 0 for the 0th-index, for it to work in the target domain.\n\nDARAIL is specifically constructed so that it learns to output 0 and works well for this artificial environment setup, because it **assumes the source trajectory to be the optimal trajectory in the target domain**. But this is a major assumption not made in the original DARC paper. In fact, I would argue (and I am open to change my mind on this if proven otherwise) that DARAIL does not work in the off-dynamics settings considered in DARC paper. For instance, consider ant which is crippled in the target domain but not the source domain (from DARC paper). Considering the source ant's trajectories as optimal could be detrimental for the target crippled ant, because most of the state transitions $(s_t, s_{t+1})$ in source domain are not even possible in the target domain. The crippled ant requires a different optimal policy to be learned, which cannot be obtained by just imitating the source state trajectories, which DARAIL would do.\n\nIn order to show that DARAIL is actually a general solution to off-dynamics RL, the paper should include experiments on:\n- All the same setups provided in DARC paper.\n- At least one more example of off-dynamics RL other than just freezing the 0-index of action.\n\n## Paper Presentation\nThe paper is often written in a confusing manner and there are several quality issues in presentation. The following need to be corrected in writing:\n- Figure 6 (b) for Ant and 6 (d) for Walker are the exact same training curves. Please fix this.\n- The paper never discusses why DARC \"fails\" is because of the 0-th action dimension being an arbitrary output in the target domain, because it was never trained to be 0 during training.\n- L2-3: \"performance degradation especially when the source domain is less comprehensive than the target domain\": \"less comprehensive\" is vague.\n- In L23-25, the paper mentions that \"in domains such as medical treatment [1] and autonomous driving [2], we cannot interact with the environment freely as the errors are too costly\". However, in Algorithm 1 L7, it assumes access to the target environment for rolling out.\n- L41-42: \"the source domain has less action support, which is normally a harder off-dynamics RL problem.\" Why is this a harder problem? If anything, target domain having a less action support would mean that the agent needs to adapt to more adverse off-dynamics conditions.\n- L64-65: \"DARAIL relaxes the assumption made in previous works that the optimal policy will receive a similar reward in both domains.\" — What does this mean, because L78 says that both the source and target domains have the same reward function? In fact, DARAIL considers the source trajectories as optimal trajectories for the target domain.\n- What does it mean for the two domains to have the same reward function, when the dynamics are different? Does the paper assume the reward function is of the form R(s) only and not R(s, a, s')? Any assumptions made in the problem setup should be clearly written.\n- In Eq 3.1 and 3.2, the product term $\\Pi_t$ should include the last terms inside the bracket.\n- L110-111: This cannot be right. The optimal policy should be proportional to the exponential of the cumulative returns, not reward at time step t.\n- Figure 1 (a): It is unclear how this figure suggests that there is any **performance degradation**. Since the dynamics are different, it is possible that the optimal reward in the target domain could be just different from the optimal reward in the source domain — comparing these two does not make sense. It only works in this paper because of the artificial formulation that the source action space is a subset of the target action space. The correct comparison would be evaluating $pi_{DARC}$ in the target domain against some $\\pi_{optimal}$ for the target domain, both in the target domain. The training reward obtained in the source domain does not tell us anything for general off-dynamics setups.\n- L151-166: This paragraph is hard to follow.\n- Figure 7: Some baselines are not trained for the same number of steps.\n\n## Method is overly complicated for the problem setup\nThe paper makes a questionable assumption that when $\\pi_{DARC}$ is optimal for source domain, the trajectories it obtains are also optimal in the target domain and thus, we should achieve similar state trajectories in the target domain. While this is a very restricted assumption, let's say for the sake of the argument, this assumption makes sense for certain useful setups. Even then, why learn $\\pi_{DARC}$ using any influence from the target trajectories at all? Why not just learn $\\pi_{src}$ from the src reward only, while disregarding the target trajectories and not applying DARC at all. And then we can transfer that policy using Gaifo to the target environment. What is the idea of doing DARC at all, when anyway the source policy is to going to be used for imitation learning from observations in the target domain? If DARC on src is really important, it should be experimentally validated with convincing arguments.\n\nQUESTIONS:\nI have listed suggestions to improve in the weakness section above. There are a lot of changes in the paper presentation that are necessary, but my major concern would still be about the limited applicability of the proposed problem setup and the method not even best suited for that proposed problem.\n\nLIMITATIONS:\nThe paper mentions a couple of limitations in the appendix section, not in the main paper. The paper does not discuss several limitations of their problem setup's applicability, their method's potential fallacies on the original DARC environments, and their limiting assumption of source trajectories being optimal for the target domain.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

comments: SUMMARY:
The paper considers the off-dynamic RL setting and identifies that existing approach, DARC, fails to obtain the true optimal policy in the target MDP. Leveraging ideas from imitation learning literature, the paper proposes a learning objective that takes into the account the dynamic shift between the source and target MDPs. The objective is theoretically sound and the conducted experiments show that the proposed method, DARAIL, is able to account for severe dynamic shift in MuJoCo environments.

STRENGTHS:
- The experiments show that DARAIL is able to account for severe dynamic shift and is not too sensitive with clipped importance sampling ratio.
- The relationship between the proposed estimator and doubly robust estimator is interesting.

WEAKNESSES:
- The notation can be defined more clearly.
	- On page 3, line 106: Is $\tau$ a trajectory sequence of state-action pairs, or just sequence of states?
		- What generates this?
	- On page 3, Eq (3.2): Is $\tau_{\pi^*}^{trg}$ Is this trajectory generated by $\pi^*$ in the target domain?
- It will be preferable if the paper can provide explanations on why DARC fails to address for the dynamic shift.

QUESTIONS:
**Questions**
- On page 4, lines 120-122: Is this always true, or just may? If I understand correctly since the dynamics are "unknown" it may recover the wrong target policy (i.e. $\pi^* \neq \pi^{trg}$)?
- DARC does not need any online learning while DARAIL does correct? I believe this should be highlighted if that is the case.
- On page 4, lines 151-152: This shared reward is an assumption correct? What if this is not true either?

**Possible typos**
- Page 2, line 128: Domain

LIMITATIONS:
**Limitations**
- The experiments are purely in MuJoCo so I am curious what can happen in other domain.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The authors introduced the approach Domain Adaptation and Reward Augmented Imitation Learning (DARAIL) for off-dynamics RL. The work aim at generating the same trajectories in the target domain as expert trajectories learned via DARC in the source domain. With GAIL-style framework and reward augmentation, DARAIL reached SOTA in bench mark off-dynamiacs environments. DRAIL does not assume the learned expert policy in the source domain to be close to the optimal policy in the target domain, and the authors are able to give a theoretical error bound of the method according to the dynamics shift scale.

STRENGTHS:
1. The problem of off-dynamics RL is very meaningful, given some target environments and data are not easily accessible.
2. The shortcoming of DARC is clearly stated. Movitation is clear.
3. The theoretical bound proof is neat.
4. Sufficient experiments on both on baseline comparison, sensitivity analysis and dynamics shift sclaes.

WEAKNESSES:
1. Typo in Line #81, expectation should be over p_trg
2. Typo in Appendix C.2 Line #537 (C.1)
3. Typo in Appendix C.2 Line #540, #546
4. There is no comparison experiements of directly using (s,s’) from the expert trejectories in the source domain for DARAIL.
5. Freezing the 0-index action is not a good way to validate the effectiveness of the method on dynamics shift. Experiemnts on ,ore diverse dynamics shift environments are needed.

QUESTIONS:
How would DARC trajectories quality influence the overall imitation learning performance in the target domain?

LIMITATIONS:
Did not test on more complicated dynamics shifts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-30