Title: Cross-Domain Policy Adaptation via Value-Guided Data Filtering
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning; Domain Adaptation; Online Dynamics Adaptation
Abstract: Generalizing policies across different domains with dynamics mismatch poses a significant challenge in reinforcement learning. For example, a robot learns the policy in a simulator, but when it is deployed in the real world, the dynamics of the environment may be different. Given the source and target domain with dynamics mismatch, we consider the online dynamics adaptation problem, in which case the agent can access sufficient source domain data while online interactions with the target domain are limited. Existing research has attempted to solve the problem from the dynamics discrepancy perspective. In this work, we reveal the limitations of these methods and explore the problem from the value difference perspective via a novel insight on the value consistency across domains. Specifically, we present the Value-Guided Data Filtering (VGDF) algorithm, which selectively shares transitions from the source domain based on the proximity of paired value targets across the two domains. Empirical results on various environments with kinematic and morphology shifts demonstrate that our method achieves superior performance compared to prior approaches.

Cross-Domain Policy Adaptation via Value-Guided Data Filtering

Abstract

Generalizing policies across different domains with dynamics mismatch poses a significant challenge in reinforcement learning. For example, a robot learns the policy in a simulator, but when it is deployed in the real world, the dynamics of the environment may be different. Given the source and target domain with dynamics mismatch, we consider the online dynamics adaptation problem, in which case the agent can access sufficient source domain data while online interactions with the target domain are limited. Existing research has attempted to solve the problem from the dynamics discrepancy perspective. In this work, we reveal the limitations of these methods and explore the problem from the value difference perspective via a novel insight on the value consistency across domains. Specifically, we present the Value-Guided Data Filtering (VGDF) algorithm, which selectively shares transitions from the source domain based on the proximity of paired value targets across the two domains. Empirical results on various environments with kinematic and morphology shifts demonstrate that our method achieves superior performance compared to prior approaches.

1 Introduction

Reinforcement Learning (RL) has demonstrated the ability to train highly effective policies with complex behaviors through extensive interactions with the environment [62, 59, 2]. However, in many situations, extensive interactions are infeasible due to the data collection costs and the potential safety hazards associated with domains such as robotics [33] and medical treatments [54]. To address the issue, one approach is to interact with a surrogate environment, such as a simulator, and then transfer the learned policy to the original domain. However, an unbiased simulator may be unavailable due to the complex system dynamics or unexpected disturbances in the target scenario, leading to a dynamics mismatch. Such a mismatch is crucial for the sim-to-real problem in robotics [1, 38, 51] and may cause performance degradation of the learned policy in the target domain. In this work, we focus on the dynamics adaptation problem, where we aim to train a well-performing policy for the target domain, given the source domain with the dynamics mismatch.

Recent research has tackled the adaptation over dynamics mismatch through various techniques, such as domain randomization [56, 53, 45], system identification [77], or simulator calibration [8], that require domain knowledge or privileged access to the physical system. Other methods have explored

Source

Domain

Source

Domain

Source

Domain

Source

Domain

Target

Domain

RL

Train

Manipulable

Source Domain

Target

Domain

RL

Train

Deployment

Limited data

Calibration

Deployment

RL

Target

Domain

Source

Domain

Expert / Offline

Data

Train

Deployment

(a) Dynamics Randomization/

System Identification/Meta RL

(b) Dynamics Adaptation with

Demonstration/Offline Dataset

(c) Dynamics Adaptation with a

Manipulable Simulator

(d) Online Dynamics Adaptation

Source

Domain

RL

Train

Target

Domain

Deployment

Limited Online

Interactions

Source Domain

Offline Data

the adaptation problem in specific scenarios, such as those with expert demonstrations [41, 32] or offline datasets [42, 49], while the effectiveness of these methods heavily depends on the optimality of demonstrations or the quality of the datasets. In contrast to these works, we consider a more general setting called online dynamics adaptation , where the agent can access sufficient source domain data and a limited number of online interactions with the target domain. We compare the settings for the dynamics adaptation problem in Figure 1.

To address the online dynamics adaptation problem, prior works mainly focus on the single-step dynamics discrepancy and practically eliminating the gap via different ways [17, 14]. However, we empirically demonstrate the limitation of the methods through a motivation example, suggesting their effectiveness heavily relies on strong assumptions about the transferability of paired domains. Theoretically, we formulate the performance bound of the learned policy with respect to the dynamics discrepancy term, which provides an explicit interpretation of the results. To address the problem, we focus on the value discrepancy between paired transitions across domains, motivated by the key idea: the transitions with consistent value targets can be seen as equivalent for policy adaptation . Based on the insight, we proposed a simple yet efficient algorithm called Value-Guided Data Filtering (VGDF) for online dynamics adaptation via selective data sharing. Specifically, we use a learned target domain dynamics model to obtain paired transitions based on the source domain state-action pair. The transitions are shared from the source to the target domain only if the value targets of the imagined target domain transition and that of the source domain transition are close. Compared to previous methods that utilize the single-step dynamics gap, our method measures value discrepancies to capture long-term differences between two domains for better adaptation.

Our contributions can be summarized as follows: 1) We reveal the limitations of prior dynamics-based methods and propose the value discrepancy perspective with theoretical analysis. 2) To provide a practical instantiation, we propose VGDF for online dynamics adaptation via selective data sharing. 3) We extend VGDF to a more practical setting with an offline source domain dataset and propose a variant algorithm motivated by novel theoretical results. 4) We empirically demonstrate the superior performance of our method given significant dynamics shifts, including kinematics and morphology mismatch, compared to previous methods.

2 Related Work

Domain adaptation in RL. Different from domain adaptation in supervised learning where different domains correspond to distinct data distributions [34], different domains in RL can differ in observation space [26], transition dynamics [56, 77, 17], embodiment [79, 43], or reward functions [16, 81, 57]. In this work, we focus on domain adaptation with dynamics discrepancies. Prior works utilizing meta RL [76, 48, 55], domain randomization [56, 53, 45], and system identification [80, 77, 15, 74] all assume the access to the distribution of training environments and rely on the hypothesis that the source and target domains are drawn from the same distribution. Another line of work has proposed to handle domain adaptation given expert demonstrations from the target domain [41, 32, 27]. These approaches align the state visitation distributions of the trained policy in the source domain to the distribution of the expert demonstrations in the target domain through state-action correspondences [79] or imitation learning [28, 21, 72]. However, near-optimal demonstrations can be challenging to acquire in some tasks. More recent works have explored the dynamics adaptation given an offline dataset collected in the target domain [42, 49], while the performance of

the trained policy depends on the quality of the dataset [50]. Orthogonal to these settings, we focus on a general paradigm where a relatively small number of online interactions with the target domain are accessible.

Online dynamics adaptation. Given limited online interactions with the target domain, several works calibrate the dynamics of the source domain by adjusting the physical parameters of the simulator [8, 58, 15, 47], while they assume the access of a manipulable simulator. Action transformation methods correct the transitions collected in the source domain by learning dynamics models of the two domains [25, 14, 78]. However, the learned model can be inaccurate, which results in model exploitation and performance degradation [30, 31]. Furthermore, the work that compensates the dynamics gap by modifying the reward function [17] is practical only if the policy that performs well in both domains exists. Instead, we do not assume the dynamics-agnostic policy exists and demonstrate the effectiveness of our method when such an assumption does not hold.

Knowledge transfer in RL. Knowledge transfer has been proposed to reuse the knowledge from other tasks to boost the training for the current task [69, 37]. The transferred knowledge can be modules ( e.g. , policy) [52, 9, 4], representations [5], and experiences [29, 39, 75, 68]. Our method is related to works transferring experiences. However, prior works focus on transferring between tasks with different reward functions instead of dynamics. When the dynamics changes, the direct adoption of commonly used temporal difference error [63] or advantage function [60] in previous works [29, 39, 68] would be inappropriate due to the shifted transition probabilities across domains. In contrast, we introduce novel measurements to evaluate the usefulness of the source domain transitions to tackle the dynamics shift problem specifically.

Theories on learning with dynamics mismatch. The performance guarantee of a policy trained with imaginary transitions from an inaccurate dynamics model has been analyzed in prior Dynastyle [64, 65, 67] model-based RL algorithms [44, 30, 61]. The theoretical results inspire us to formulate performance guarantees in the context of dynamics adaptation.

3 Preliminaries and Problem Statement

We consider two infinite-horizon Markov Decision Processes (MDP) M src := ( S , A , P src , r, γ, ρ 0 ) and M tar := ( S , A , P tar , r, γ, ρ 0 ) for the source domain and the target domain, respectively. The two domains share the same state space S , action space A , reward function r : S × A → R with range [0 , r max ] , discount factor γ ∈ [0 , 1) , and the initial state distribution ρ 0 : S → [0 , 1] . The two domains differ on the transition probabilities, i.e. , P src ( s ′ | s, a ) and P tar ( s ′ | s, a ) .

We define the probability that a policy π encounters state s at the time step t in MDP M as P π M ,t ( s ) . We denote the normalized probability that a policy π encounters state s in M as ν π M ( s ) := (1 -γ ) ∑ ∞ t =0 γ t P π M ,t ( s ) , and the normalized probability that a policy encounters state-action pair ( s, a ) in M is ρ π M ( s, a ) := (1 -γ ) ∑ ∞ t =0 γ t P π M ,t ( s ) π ( a | s ) . The performance of a policy π in M as is formally defined as η M ( π ) := E s,a ∼ ρ π M [ r ( s, a )] .

We focus on the online dynamics adaptation problem where limited online interactions with the target domain are accessible, which can be defined as follows:

Definition 3.1. (Online Dynamics Adaptation) Given source domain M src and target domain M tar with different dynamics, we assume sufficient data from the source domain (online or offline) and a relatively small number of online interactions with M tar (e.g., Γ := # source domain data # target domain data = 10 ), hoping to obtain a near-optimal policy π concerning the target domain M tar .

The prior work [17] also focuses on the online dynamics adaptation problem with online source domain interactions. The proposed algorithm DARC estimates the dynamics discrepancy via learned domain classifiers and further introduces a reward correction ( i.e. , ∆ r ( s, a, s ′ ) ≈ log ( P tar ( s ′ | s, a ) /P src ( s ′ | s, a )) ) to optimize policy together with the task reward r ( i.e. , r ( s, a ) + ∆ r ( s, a, s ′ ) ), discouraging the agent from dynamics-inconsistent behaviors in the source domain.

4 Guaranteeing Policy Performance from a Value Discrepancy Perspective

In this section, we will first present an example demonstrating the limitation of the prior method considering the dynamics discrepancy. Following that, we provide a theoretical analysis of the

Source Domain

Target Domain

Ours

DARC

Learned Q Table

State Visitation

Ours

DARC

(a)

(b)

(c)

Source

Domain

Target

Domain

dynamics-based method to provide an interpretation of the experiment results. Finally, we introduce a novel perspective on value discrepancies across domains for the online dynamics adaptation problem.

4.1 Motivation Example

We start with a 2D grid world task shown in Figure 2 (a), where the agent represented by the red dot needs to navigate to the green square representing the goal. We design source and target domains with different layouts and train a policy to reach the goal successfully in the target domain. We investigate the performance of DARC [17] that trains the policy with dynamics-guided reward correction and our proposed method (Section 5), using tabular Q -learning [73] as the backbone for all methods. Detailed environment settings are shown in Appendix D.

As the empirical state visitations and the learned Q tables show in Figure 2, DARC is stuck in the room and fails to obtain near-optimal Q -values, leading to poor performance. Specifically, we circle out four positions where specific actions will lead to the states with a dynamics mismatch concerning the two domains. Due to the introduced reward correction on the source domain transitions with dynamics mismatch, DARC learns overly pessimistic value estimations of particular state-action pairs, which hinders the agent from the optimal trajectory concerning the target domain. However, the values of the following inconsistent states, induced by the particular state-action pairs, are not significantly different concerning the target domain. The value difference quantifies the discrepancy of the long-term behaviors rather than single-step dynamics. Motivated by the value discrepancy perspective, our proposed method (Section 5.1) demonstrates superior performance.

4.2 Theoretical Interpretations and Value Discrepancy Perspective

To provide rigorous interpretations for the results, we derive a performance guarantee for the dynamicsguided methods, which mainly build on the theories proposed in prior methods [30, 17].

Theorem 4.1. (Performance bound controlled by dynamics discrepancy.) Denote the source domain and target domain with different dynamics as M src and M tar , respectively. We have the performance difference of any policy π evaluated under M src and M tar be bounded as below,

The proof of Theorem 4.1 is given in Appendix B. We observe that the derived performance bound in (1) is controlled by the dynamics discrepancy term (a). Intuitively, the performance difference would be minor when the dynamics discrepancy between the two domains is negligible. DARC [17] applies

the Pinsker's inequality [13] and derives the following form:

Based on the result in (2), DARC optimizes the policy by converting the second term in RHS to a reward correction ( i.e. , ∆ r := log( P tar ( s ′ | s, a ) /P src ( s ′ | s, a )) ), leading to the dynamics discrepancybased adaptation. However, given the transition from the source domain ( i.e. , P src ( s ′ | s, a ) ≈ 1 ), the reward correction will lead to significant penalty ( i.e. , log( P tar ( s ′ | s, a ) /P src ( s ′ | s, a )) ≪ 0 ) if the likelihood estimation of the transition concerning the target domain is low ( i.e. , P tar ( s ′ | s, a ) ≈ 0 ). Consequently, the value estimation of the transition with dynamics mismatch tends to be overly pessimistic as shown in Figure 2 (c), which hinders learning an effective policy concerning the target domain.

Instead of myopically considering the single-step dynamics mismatch, we claim that the transitions with significant dynamics mismatch can be equivalent concerning the value estimations that evaluate the long-term behaviors. Due to the dynamics shift across domains, a state-action pair ( i.e. , ( s, a ) ) would lead to two different next-states ( i.e. , s ′ src , s ′ tar ), the paired transitions are nearly equivalent for temporal different learning if the induced value estimations are close ( i.e. , | V ( s ′ src ) -V ( s ′ tar ) | ≤ ϵ ). Motivated by this, we derive a performance guarantee from the value difference perspective.

Theorem 4.2. (Performance bound controlled by value difference.) Denote source domain and target domain as M src and M tar , respectively. We have the performance guarantee of any policy π over the two MDPs:

The proof of Theorem 4.2 is given in Appendix B. The value difference term provides a novel perspective: the performance can be guaranteed if the transitions from the source domain lead to consistent value targets in the target domain . The result further highlights the value consistency perspective for the online dynamics adaptation problem.

5 Value-Guided Data Filtering

In this section, we propose Value-Guided Data Filtering (VGDF), a simple yet efficient algorithm for online domain adaptation via selective data sharing. Then we introduce the setting with offline source domain data and a variant of VGDF based on novel theoretical results. The pseudocodes are shown in Appendix A, and the illustration of VGDF is shown in Figure 3.

5.1 Dynamics Adaptation by Selective Data Sharing

Inspired by the performance bound proposed in Theorem 4.2, we can guarantee the policy performance by controlling the value difference term in (3). As discussed in Section 4.2, the paired transitions concerning two domains, induced by the same state-action pair, can be regarded as equivalent for temporal difference learning when the corresponding values are close. Thus, we propose to select source domain transitions with minor value discrepancies for dynamics adaptation.

To select rational transitions from the source domain, we need to compare the value differences of paired transitions based on the same source domain state-action pair ( s src , a src ) . Formally, given a state-action pair ( s src , a src ) from the source domain, our objective is to estimate whether the value-difference between s ′ tar and s ′ src is sufficiently small, i.e. ,

where s ′ tar ∼ P tar ( ·| s src , a src ) , s ′ src ∼ P src ( ·| s src , a src ) , 1 denotes the indicator function and ϵ can be a predefined threshold.

To obtain ∆( s src , a src ) , we need to perform policy evaluation over the states to obtain the value estimations given the paired next states ( i.e. , s ′ src , s ′ tar ), as formulated in Eq. (4). Monte Carlo (MC)

Source Domain

Target Domain

!

!"#

Data

Filtering

RL

!

!"#

~

(#, %, &, #

!"#

$

)

(

%

&

' (

!"#

$

%

{*

!"#

&

}

%

!

!"#

$

(#

%#&

'

)

,

ℙ ⋅ Mean {,

)*+

,

}

-

)*+

,

}

-

Λ

(#

!"#

$

)

(#, %)

Rejection

Sampling

!

67"

evaluation can provide unbiased values by rolling the policy starting from specific states [66]. However, since the environment is not manipulable, we cannot perform MC evaluation from arbitrary states. Thus, we propose to use an estimated value function for policy evaluation. In this work, we adopt the Fitted Q Evaluation (FQE) [46] that is widely used in off-policy RL algorithms [40, 23, 24]. Specifically, we utilize a learned Q function Q θ : S × A → R for evaluation.

Furthermore, one problem is that the corresponding target domain next state s ′ tar induced by ( s src , a src ) is unavailable in practice. To achieve this, we train a dynamics model with the collected data from the target domain. Following prior works [36, 10], we employ an ensemble of Gaussian dynamics models { T ϕ i ( s ′ | s, a ) } i M =1 , in an attempt to capture the epistemic uncertainty due to the insufficient target domain samples. Given the source domain state-action pair ( s src , a src ) , we generate an ensemble of fictitious states and obtain the corresponding values for each state-action pair, which we call fictitious value ensemble (FVE) Q π tar ( s src , a src ) :

In practice, the choice of ϵ in Eq. (4) is also nontrivial due to task-specific scales of the values and the non-stationary value function during training. We replace the absolute value difference with the likelihood estimation to address the problem. Specifically, we construct a Gaussian distribution with the mean and variance of FVE denoted as N (Mean( Q π tar ( s src , a src )) , Var( Q π tar ( s src , a src ))) . Estimating the value of the source domain state as V π tar ( s ′ src ) := Q θ ( s ′ src , a ′ src ) | a ′ src ∼ π ( ·| s ′ src ) , we introduce Fictitious Value Proximity (FVP) representing the likelihood of the source domain state value in the distribution:

Based on the likelihood estimation, we utilize the rejection sampling to select fixed percentage data ( i.e. , 25% ) with the highest likelihood from a batch of source domain transitions at each training iteration. Specifically, we train the value function by optimizing the following objective:

Λ ξ % is the top ξ -quantile likelihood estimation of the minibatch sampled from source domain data, T represents the Bellman operator, and D src , D tar denote replay buffers of two domains.

Consider the case when the agent can perform online interactions with the source domain, the training data mostly comes from the source domain, while we aim to train a policy for the target domain. Hence, exploring the source domain is essential to collect transitions that might be high-value concerning the target domain. Thus, we introduce an exploration policy π E that maximizes the approximate upper confidence bound of the Q -value, i.e. , π E ← arg max π E E s ∼ D tar ∪ D src [ Q UB ( s, a ) | a ∼ π E ( ·| s ) ] , where Q UB ( s, a ) := max { Q θ i ( s, a ) } 2 i =1 under the implementation with SAC [24] backbone. Importantly, the exploration policy π E is separate from the main policy π learned via vanilla SAC. π E and π are used for data collection in the source domain and target domain, respectively. The optimistic data collection technique has been proposed for advanced exploration [11] while we utilize the technique in online dynamics adaptation setting.

5.2 Adaptation with Offline Dataset of Source Domain

So far, we have discussed the setting where the agent can interact with the source domain to collect data actively. Nonetheless, simultaneous online access to the source and target domain might sometimes be impractical. In order to address the limitation, we aim to extend our method to the setting we refer to as Offline Source with Online Target , in which the agent can access a source domain offline dataset and a relatively small number of online interactions with the target domain.

To adapt VGDF to such a setting, we propose a novel theoretical result of the performance guarantee:

Theorem 5.1. Under the setting with offline source domain dataset D whose empirical estimation of the data collection policy is π D ( a | s ) := ∑ D 1 ( s,a ) ∑ D 1 ( s ) , let M src and M tar denote the source and target domain, respectively. We have the performance guarantee of any policy π over the two MDPs:

The proof of Theorem 5.1 is given in Appendix B. This theorem highlights the importance of policy regularization and value difference for achieving desirable performance. It is worth noting that the policy regularization term can shed light on the impact of behavior cloning, which has been proven effective for offline RL [22]. Additionally, the value difference term has a similar structure to that of Theorem 3. Thus, we propose a variant called VGDF + BC that combines behavior cloning loss with the original selective data sharing scheme. The pseudocode is shown in Algorithm 2, Appendix A.

6 Experiments

In this section, we present empirical investigations of our approach. We examine the effectiveness of our method in scenarios with various dynamics shifts, including kinematic change and morphology change. Furthermore, we provide ablation studies and qualitative analysis of our method. Details of environment settings and the implementation are shown in Appendix D and Appendix E, respectively. Additional results are in Appendix F.

6.1 Adaptation Performance Evaluation

To systematically investigate the adaptation performance of the methods, we construct two types of dynamics shift scenarios, including kinematic shift and morphology shifts, for four environments ( HalfCheetah , Ant , Walker , Hopper ) from Gym Mujoco [71, 7]. We use the original environment as the source domain across all experiments. To simulate kinematic shifts, we limit the rotation angle range of specific joints to simulate the broken joint scenario. As for morphology shifts, we modify the size of specific limbs while the number of limbs keeps unchanged to ensure the state/action space consistent across domains. Full details of the environment settings are deferred to Appendix D.

We compare our algorithm with four baselines: (i) DARC [17] trains the domain classifiers to compensate the agent with an extra reward for seeking dynamics-consistent behaviors; (ii) GARAT [14] trains the policy with an adversarial imitation reward in the grounded source domain via action transformation [25]; (iii) IW Clip (Importance Weighting Clip) performs importance-weighted bellman updates for source domain samples. The importance weights ( i.e. , P tar ( s ′ | s, a ) /P src ( s ′ | s, a ) ) are approximated by the domain classifiers proposed in DARC, and we clip the weight to [10 -4 , 1] to stabilize training; (iv) Finetune uses the 10 5 target domain transitions to finetune the policy trained in the source domain with 1M samples. Furthermore, Zero-shot shows the performance of directly transferring the learned policy in the source domain to the target domain, and Oracle demonstrates the performance of the policy trained in the target domain from scratch with 1M transitions. We run all algorithms with the same five random seeds. The implementation details are given in Appendix E.1.

As the results in Figure 4 show, our method outperforms GARAT and IW Clip in all environments. DARC demonstrates competitive performance only in the first two environments, while it does not work in other environments. We believe that the assumption of DARC does not hold in the failure cases due to the significant dynamics mismatch. GARAT fails in almost all environments, which we

ck thigh

3500

3000

2500

2000

1500

1000

Walker-bro

00

00

00

00

Hopper-broke

3000

2500

2000

1500

HaifCheetan-bro

5000

4000

√

.0

Ant-shor

4000

3000

2000

0.0

Hopper

3000

2500

2000

Walker-

00

00

HalfCheetah

000

000

000

believe is caused by the impractical action transformation from inaccurate dynamics models. The performance of Zero-shot suggests that the policies trained in the source domains barely work in the target domains due to dynamics mismatch. Finetune achieves promising results and outperforms our method in two of eight environments. We believe that the temporally-extended behaviors of the pre-trained policy benefit learning in the downstream tasks with the assistance of efficient exploration. Nonetheless, our method is the only one that outperforms or matches the asymptotic performance of Oracle in four out of eight environments.

6.2 Ablation Studies

To investigate the impact of design components in our method, we perform ablation analysis on the ratio of transitions Γ , data selection ratio ξ % , and the optimistic exploration.

Data ratio Γ . We employ different ratios of transitions from the source domain versus those from the target domain ( Γ = 5 , 10 , 20 ) for variants of our algorithm. The results shown in Figure 5 demonstrate that the performance of our algorithm improves with more source domain transitions when the number of target domain transitions is the same. This finding indicates that VGDF can fully exploit the reusable

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

5000

Return

HalfCheetah - broken back thigh

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

HalfCheetah - no thighs

=5

=10

=20

source domain transitions to enhance the training efficiency concerning the target domain.

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

5000

Return

HalfCheetah - broken back thigh

w/

E

w/o

E

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

Return

HalfCheetah - no thighs

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

Return

Hopper - broken joints

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

Return

Hopper - big head

Data selection ratio ξ % . We employ different data ratios ( 10% , 25% , 50% , 75% ) for the variants of our algorithm. Furthermore, we propose a baseline algorithm Mix that learns with all source domain samples without selection ( ω ( s, a, s ′ ) ≡ 1 in Eq. (7)). The results, shown in Figure 6, indicate

that our algorithm performs robustly under various ratios within a specific range ( e.g. , ξ % ≤ 50% ). Surprisingly, Mix performs exceptionally well in environments with kinematic mismatches but fails in scenarios with morphology shifts. We attribute this to the less significant dynamics shift induced by kinematic changes compared to morphology changes.

Optimistic data collection. To validate the effect of the optimistic exploration π E , we introduce a variant of our method without π E . The results are shown in Figure 7. Removing the optimistic exploration technique results in performance degradation in three out of four environments concerning the sample efficiency, validating the effectiveness of the exploration policy.

6.3 Performance under Offline Source with Online Target

In this subsection, we extend our method to the setting with a source domain offline dataset and limited online interactions with the target domain, investigating the performance of our method without online access to the source domain. We use the D4RL medium datasets [20] of three environments ( i.e. , HalfCheetah , Walker , Hopper ) for evaluation. We compare the proposed VGDF + BC (Section 5.2) with the following baselines: Offline only that directly transfers the offline learned policy via CQL [35] to the target domain; Symmetric sampling [3] that samples 50% of the data from the target domain replay buffer and the remaining 50% from the source domain offline dataset for each training step; H2O [49] that penalizes the Q function learning on source domain transitions with the estimated dynamics gap via learned classifiers. All algorithms have limited interactions with the target domain to 10 5 steps. The experimental details are shown in Appendix E.2. The results shown in Table 1 demonstrate that our method outperforms the other methods in four out of six environments, indicating that filtering the source domain data with the value consistency paradigm is effective in the offline-online setting.

6.4 Quantifying Dynamics Mismatch via Fictitious Value Proximity

Although the empirical results suggest that our method can adapt the policy in the face of various dynamics shifts, the degree of the dynamics mismatch can only be evaluated via the adaptation performance rather than be quantified directly. Here, we propose quantifying the dynamics shifts via the proposed Fictitious Value Proximity (FVP) (Section 5.1).

We approximate the FVP in Eq. (5) by calculating the average likelihood of a batch of samples from the source domain by E [Λ( s, a, s ′ )] ≈ 1 B ∑ ( s,a,s ′ ) ˆ Λ( s, a, s ′ ) . We show the approximated FVP in Ant environments with kinematic or morphology shifts in Figure 8. We observe a significant gap between the FVP values of the paired domains, which suggests the target domain with the morphology shifts is "closer" to the source domain than the target domain with the kinematic shifts with respect to the value difference. FVP measured by value differences quantifies the long-term effect on the expected return. Such a measurement can be regarded as a way to quantify the domain discrepancies.

Kinematic Shift

Morphology Shift

7 Conclusion

This work addresses the online dynamics adaptation problem by proposing VGDF that selectively shares the source domain transitions from a value consistency paradigm. Starting from the motivation example, we reveal the limitation of the prior dynamics-based method. Then we introduce a novel value discrepancy perspective with theoretical analysis, motivated by the insight that paired transitions with consistent value targets can be regarded as equivalent for training. Practically, we propose VGDF and the variant for the offline source domain setting. Empirical studies demonstrate the effectiveness of our method under significant dynamics gaps, including kinematics shifts and morphology shifts.

Limitation and future directions. One limitation of our method is the reliance on the ensemble dynamics models. However, the recent work estimating the epistemic uncertainty with a single model [19] could be applicable. Furthermore, value-aware model learning [18] may improve our method by training dynamics models with accurate value predictions of the generated samples. Exploring the effectiveness of value consistency for generalizing across reward functions can be another direction for future research. Finally, validating the effectiveness of the data sharing method in the Sim2Real problem would contribute to the robotics community. The online interaction with the reality system could be risky, recent works [6, 70] can be integrated for safe online interactions.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No.62306242), the National Key R&D Program of China (Grant No.2022ZD0160100), Shanghai Artificial Intelligence Laboratory, Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103), Scientific Research Development Center in Higher Education Institutions by the Ministry of Education, China (No.2021ITA10013), Shanghai Engineering Research Center of Al and Robotics, Engineering Research Center of AI and Robotics, Ministry of Education, China. We would like to thank the anonymous reviewers for their valuable suggestions.

References

Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019.

Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang. Principled exploration via optimistic bootstrapping and backward induction. In International Conference on Machine Learning , pages 577-587. PMLR, 2021.

Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02948 , 2023.

Mohammadamin Barekatain, Ryo Yonetani, and Masashi Hamaya. Multipolar: multi-source policy aggregation for transfer reinforcement learning between diverse environmental dynamics. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pages 3108-3116, 2021.

Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning , pages 501-510. PMLR, 2018.

Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. In International Conference on Learning Representations , 2020.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real

world experience. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8973-8979. IEEE, 2019.

Ching-An Cheng, Andrey Kolobov, and Alekh Agarwal. Policy improvement via imitation of multiple oracles. Advances in Neural Information Processing Systems , 33:5587-5598, 2020.

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems , 31, 2018.

Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems , 32, 2019.

Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.org , 2016-2021.

Imre Csiszár and János Körner. Information theory: coding theorems for discrete memoryless systems . Cambridge University Press, 2011.

Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems , 33:3917-3929, 2020.

Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto-tuned sim-to-real transfer. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 1290-1296. IEEE, 2021.

Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 , 2016.

Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In International Conference on Learning Representations , 2020.

Amir-massoud Farahmand, Andre Barreto, and Daniel Nikovski. Value-aware loss function for model-based reinforcement learning. In Artificial Intelligence and Statistics , pages 1486-1494. PMLR, 2017.

Angelos Filos, Eszter Vértes, Zita Marinho, Gregory Farquhar, Diana Borsa, Abram Friesen, Feryal Behbahani, Tom Schaul, André Barreto, and Simon Osindero. Model-value inconsistency as a signal for epistemic uncertainty. arXiv preprint arXiv:2112.04153 , 2021.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations , 2018.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Josiah P Hanna and Peter Stone. Grounded action transformation for robot learning in simulation. In Thirty-first AAAI conference on artificial intelligence , 2017.

Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. In International Conference on Learning Representations , 2020.

Donald Hejna, Lerrel Pinto, and Pieter Abbeel. Hierarchically decoupled imitation for morphological transfer. In International Conference on Machine Learning , pages 4159-4171. PMLR, 2020.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems , 32, 2019.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems , 33:21810-21823, 2020.

Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning , pages 5286-5295. PMLR, 2020.

Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274, 2013.

Wouter M Kouw and Marco Loog. A review of domain adaptation without target labels. IEEE transactions on pattern analysis and machine intelligence , 43(3):766-785, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:11791191, 2020.

Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations , 2018.

Alessandro Lazaric. Transfer in reinforcement learning: a framework and a survey. In Reinforcement Learning , pages 143-173. Springer, 2012.

Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics , 5(47):eabc5986, 2020.

Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. Advances in neural information processing systems , 33:7754-7767, 2020.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster) , 2016.

Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-based imitation learning. In International Conference on Learning Representations , 2019.

Jinxin Liu, Zhang Hongyin, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. In International Conference on Learning Representations , 2021.

Xingyu Liu, Deepak Pathak, and Kris M Kitani. Revolver: Continuous evolutionary models for robot-to-robot policy transfer. arXiv preprint arXiv:2202.05244 , 2022.

Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations , 2018.

Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning , pages 1162-1176. PMLR, 2020.

Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

Fabio Muratore, Christian Eilers, Michael Gienger, and Jan Peters. Data-efficient domain randomization with bayesian optimization. IEEE Robotics and Automation Letters , 6(2):911918, 2021.

Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through metareinforcement learning. In International Conference on Learning Representations , 2018.

Haoyi Niu, Shubham Sharma, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, and Xianyuan Zhan. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. arXiv preprint arXiv:2206.13464 , 2022.

Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems , 34:2328323295, 2021.

Michael O'Connell, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics , 7(66):eabm6597, 2022.

Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In ICLR (Poster) , 2016.

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3803-3810. IEEE, 2018.

Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference , pages 147-163. PMLR, 2017.

Roberta Raileanu, Max Goldstein, Arthur Szlam, and Rob Fergus. Fast adaptation to new environments via policy-dynamics value functions. In Proceedings of the 37th International Conference on Machine Learning , pages 7920-7931, 2020.

Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283 , 2016.

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning , pages 5331-5340. PMLR, 2019.

Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators. arXiv preprint arXiv:1906.01728 , 2019.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

Jian Shen, Han Zhao, Weinan Zhang, and Yong Yu. Model-based policy optimization with unsupervised model adaptation. Advances in Neural Information Processing Systems , 33:28232834, 2020.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815 , 2017.

Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3(1):9-44, 1988.

Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990 , pages 216-224. Elsevier, 1990.

Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin , 2(4):160-163, 1991.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Richard S Sutton, Csaba Szepesvári, Alborz Geramifard, and Michael Bowling. Dyna-style planning with linear function approximation and prioritized sweeping. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence , pages 528-536, 2008.

Yunzhe Tao, Sahika Genc, Jonathan Chung, Tao Sun, and Sunil Mallya. Repaint: Knowledge transfer in deep reinforcement learning. In International Conference on Machine Learning , pages 10141-10152. PMLR, 2021.

Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research , 10(7), 2009.

Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems , 34:13859-13869, 2021.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158 , 2018.

Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8(3):279-292, 1992.

Annie Xie, Shagun Sodhani, Chelsea Finn, Joelle Pineau, and Amy Zhang. Robust policy learning over multiple uncertainty sets. arXiv preprint arXiv:2202.07013 , 2022.

Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems , 34:11501-11516, 2021.

Wenhao Yu, C Karen Liu, and Greg Turk. Policy transfer with strategy optimization. In International Conference on Learning Representations , 2018.

Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. arXiv preprint arXiv:1702.02453 , 2017.

Grace Zhang, Linghan Zhong, Youngwoon Lee, and Joseph J Lim. Policy transfer across visual and dynamics domain gaps via iterative grounding. arXiv preprint arXiv:2107.00339 , 2021.

Qiang Zhang, Tete Xiao, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Learning crossdomain correspondence for control with dynamics cycle-consistency. In International Conference on Learning Representations , 2020.

Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment probing interaction policies. In International Conference on Learning Representations , 2018.

Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via metalearning. In International Conference on Learning Representations , 2019.

A Algorithm Description

The pseudocode of VGDF is presented in Algorithm 1. We utilize SAC [24] as our backbone algorithm. We employ a fixed entropy temperature coefficient in all experiments, demonstrating sufficient empirical performance. The training of the dynamics model ensemble follows prior works [10, 30] with the MLE loss. The calculation of the Fictitious Value Proximity follows Eq. (6) proposed in Section 5.1. Furthermore, the pseudocode of VGDF + BC is presented in Algorithm 2. We introduce the value-normalized tradeoff between the behavior cloning loss and the policy gradient following the prior work [22].

Algorithm 1 Value-Guided Data Filtering (VGDF)

Input: Source domain M src , target domain M tar , and transition ratio Γ (= 10) (source vs. target). Initialization: Policy π , exploration policy π E , value functions { Q θ i } i =1 , 2 , replay buffers { D src , D tar } , dynamics model ensemble { T ϕ i } i M =1 , data selection ratio ξ , batch size B , entropy temperature coefficient λ .

B Proofs of the Performance Guarantees

This section presents the proof of our main results. Specifically, we propose that the value discrepancy can be leveraged for the performance guarantee across different domains Lemma C.3. In Theorem B.1, we convert the performance bound induced by the value discrepancy into a novel form for the offline source domain setting.

Theorem B.1. (Performance bound controlled by dynamics discrepancy.) Denote the source domain and target domain with different dynamics as M src and M tar , respectively. We have the

Algorithm 2 Value-Guided Data Filtering + Behavior Cloning (VGDF + BC)

Input: Source domain offline dataset D src , target domain M tar , max interaction steps with the target domain T max , and transition ratio Γ (:= | D src | T = 10) (source vs. target).

max Initialization: Policy π , value functions { Q θ i } i =1 , 2 , target domain replay buffer D tar , dynamics model ensemble { T ϕ i } i M =1 , data selection ratio ξ , batch size B , entropy temperature coefficient λ , train repeat K , behavior cloning constant α .

2: # Interact with the target domain

3: Sample transition ( s tar , a tar , r tar , s ′ tar ) using π in M tar

5: # Repeat training for K times per step

7: Optimize dynamics ensemble { T ϕ i } i M =1 with D tar via Eq. (13)

10: Obtain Fictitious Value Proximity (FVP) { Λ( s, a, s ′ ) } B via Eq. (6) for transitions in b src

12: # Optimize value function with data filtering

14:

15: # Optimize policy with behavior cloning regularization

17:

π

19: end for

π

β

2

B

b

tar

∪

b

src

1

20:

end for performance difference of any policy π evaluated under M src and M tar be bounded as below,

Proof. We have

∑

[

min

{

Q

(

s, a

)

, Q

(

s, a

)

}

+

θ

θ

2

∼

a

π

(

·|

s

)

λ

H

[

π

]

]

-

Theorem B.2. (Performance bound controlled by value difference.) Denote the source domain and target domain as M src and M tar , respectively. We have the performance guarantee of any policy π over the two MDPs:

Proof. We have

Theorem B.3. Under the setting with offline source domain dataset D whose empirical estimation of the data collection policy is π D ( a | s ) := ∑ D 1 ( s,a ) ∑ D 1 ( s ) , let M src and M tar denote the source and target domain, respectively. We have the performance guarantee of any policy π over the two MDPs:

Proof. We have

We have

Combining the two inequalities above completes the proof.

C Proofs of Lemmas

This section provides proof of several lemmas used for our theoretical results. The first lemma is adopted from [44], and the proof is essentially the same as the original paper. Lemma C.2 and Lemma C.3 support the derivation of the performance difference bound in Theorem B.3.

Lemma C.1. (Telescoping Lemma, Lemma 4.3 in [44].) Let M 1 := ( S , A , P 1 , r, γ ) and M 2 := ( S , A , P 2 , r, γ ) be two MDPs with different dynamics P 1 and P 2 . Given a policy π , let

we have

Proof. Define W j as the expected return when executing π on M 1 for the first j steps, then switching to π and M 2 for the remainder. That is

Then we have

Thus we can obtain

Convert W j and W j +1 as following:

Plug back to Eq.11 and we obtain

Lemma C.2. (Extension of Telescoping Lemma.) Let M 1 := ( S , A , P 1 , r, γ ) and M 2 := ( S , A , P 2 , r, γ ) be two MDPs with different dynamics P 1 and P 2 . Given two policies π 1 , π 2 , let

we have

Proof. Define W j as the expected return when executing π 1 on M 1 for the first j steps, then switching to π 2 and M 2 for the remainder. That is

Then we have

Thus we can obtain

Convert W j and W j +1 as following:

Plug back to Eq.12 and we obtain

Lemma C.3. (Bound of G π 1 ,π 2 M 1 , M 2 ( s, a ) .) Let

we have

Proof. We have

For ( a ) , we have

For ( b ) , we have

Adding these two bounds together yields the desired result.

D Detailed Environment Setting

D.1 Grid World

In the grid world environment, the agent obtains the X-Y coordination as the state and executes one of the four actions (Up, Down, Left, Right) at each time step. A non-zero reward 1 . 0 is provided only if the agent reaches the goal. Each episode terminates when the agent reaches the goal or the episode length of 256 is reached. The source domain and the target domain of the grid world are shown in Figure 9. For each algorithm, the agent interacts with the source and target domains for 5 e 5 and 5 e 4 steps, respectively.

D.2 Mujoco Environments

To investigate the performance of the algorithm thoroughly, we design eight environments based on four Mujoco [71] benchmarks from Gym [7] including HalfCheetah-v2, Ant-v4, Walker2D-v2, and Hopper-v2. For each benchmark, we propose two variants with kinematic shift or morphology shift. We run all experiments with the original environment as the source domain and the variation environment as the target domain. Detailed modifications of the environments are shown below, and the illustration of the environments is shown in Figure 10. For algorithms that access interactions with both domains, the agent interacts with the source and target domains for 10 6 and 10 5 steps, respectively.

Detailed modifications of the environments with kinematic shifts are shown below:

HalfCheetah - broken back thigh : We modify the rotation range of the joint on the thigh of the back leg from [ -0 . 52 , 1 . 05] to [ -0 . 0052 , 0 . 0105] .

Source Domains

Kinematic shifts

Morphology shifts

KX★

一

Ant - broken hips : We modify the rotation range of the joints on the hip of leg 1 and leg 2 from [ -30 , 30] to [ -0 . 3 , 0 . 3] .

Walker - broken right foot : We modify the rotation range of the joint on the foot of the right leg from [ -45 , 45] to [ -0 . 45 , 0 . 45] .

Hopper - broken joints : We modify the rotation range of the joint on the head from [ -150 , 0] to [ -0 . 15 , 0] and the joint on foot from [ -45 , 45] to [ -18 , 18] .

Detailed modifications of the environments with morphology shifts are shown below:

HalfCheetah - no thighs : We modify the size of both thighs. Detailed modifications of the xml file are:

Ant - short feet : We modify the size of feet on leg 1 and leg 2. Detailed modifications of the xml file are:

Walker - no right thigh : We modify the size of thigh on the right leg. Detailed modifications of the xml file are:

Hopper - big head : We modify the size of the head. Detailed modifications of the xml file are:

E Algorithms and Implementation Details

E.1 Implementation Details

The details of our algorithm and baseline methods are specified as follows:

SAC : We first specify the implementation of the shared backbone algorithm SAC utilized in all algorithms. The policy and the value function are two-layer MLP with 256 hidden units using ReLU activation. The learning rate is 3 e -4 . Discount γ is set as 0 . 99 in all environments. The temperature coefficient is fixed as 0 . 2 . The batch size is 128 . The smoothing coefficient of the target networks is 0 . 005 . The training delay of the policy is set as 2 . The replay buffer size is 1 e 6 .

VGDF : We use a five-layer MLP with 200 units as the dynamics model using Swish activation following prior works [10, 30]. The ensemble size is 7 . We set the data selection ratio ξ % as 25% in the experiments shown in Section 6.1. For each probabilistic dynamics model T ϕ i ( s t +1 , r t | s t , a t ) = N ( µ ϕ i ( s t , a t ) , Σ ϕ i ( s t , a t )) , i = 1 , . . . , M , we train the model by maximizing the objective:

The exploration policy is a two-layer MLP with 256 hidden units. We warm-start the algorithm by utilizing samples from both domains without selection for the first 1 e 5 steps in the source domain.

DARC : We follow the default configurations of the public implementation ( https://github. com/google-research/google-research/tree/master/darc ). The domain classifiers q ψ SAS ( s t , a t , s t +1 ) , q ψ SA ( s t , a t ) are trained by maximizing the cross-entropy losses:

Following the original implementation, we use the standard Gaussian noise for the domain classifier training. During training, a reward correction ∆ r ( s t , a t ) is augmented to the original reward r ( s t , a t ) of each source domain transition, i.e. ˜ r ( s t , a t ) := r ( s t , a t ) + ∆ r ( s t , a t ) . The reward correction is calculated by:

We warm-start the algorithm by training with samples from both domains for the first 10 5 steps following the original implementation.

GARAT : We use the author implementation with default configurations ( Supplemental in https://proceedings.neurips.cc/paper/2020/hash/ 28f248e9279ac845995c4e9f8af35c2b-Abstract.html ). We add the XML files of our customized environments to rl_gat/envs/assets/ folder. We limit the extra interactions with the grounded source environments as 10 5 for fair comparisons with other algorithms.

Importance Weighting Clip (IW Clip) : We use the domain classifiers same as DARC to calculate the importance weight w ( s, a, s ′ ) . The importance weighting is calculated by:

where q ψ SAS and q ψ SA are the domain classifiers proposed in [17]. We use the importance weighing to reweight the value training with source domain samples. Specifically,

To stabilize training, we clip the importance weight between [1 e -4 , 1] , same as the prior work [49].

Finetune : We first train a policy in the source domain with 10 6 steps. Then we transfer the policy to the target domain and further train the policy for 10 5 steps.

The detailed hyperparameters of all algorithms are listed in Table. 2, and we use the same hyperparameters across all environments.

E.2 Implementation Details of the Offline-Online Experiments

To evaluate the performance of our algorithm in the offline source online target setting, we use medium datasets from D4RL [20] for three environments ( i.e. , HalfCheetah, Hopper, Walker). We use the same source domain offline dataset for each environment's two different target domains. For the algorithms performing online learning using offline data ( i.e. , Symmetric sampling , H2O , VGDF + BC ), we perform the online interactions with the target domain for 10 5 steps and use 10 6 source domain transitions, the training is repeated for 10 times per step in the target domain. The details of the methods are specified as follows:

Offline only : We directly transfer the policy learned through CQL [35] with the source domain offline dataset. For the CQL implementation, we follow the suggested configurations in a public CQL implementation ( https://github.com/tinkoff-ai/CORL ). We perform training for 10 6 steps with the offline dataset and report the zero-shot performance of the learned policy in the target domain.

Symmetric sampling [3]: We perform the value function training by combining CQL optimization (with offline transitions) and SAC optimization (with online transitions). For each training step, we

sample 50% of the data from the target domain replay buffer and the remaining 50% from the source domain offline dataset. The CQL and SAC loss is computed with the corresponding transitions.

H2O [49]: We follow the original implementation that learns the classifiers to estimate the dynamics discrepancy across domains and perform the clipped importance weighting on the CQL loss on the source domain data. Same as Symmetric sampling , we repeat the training for 10 times per step in the target domain.

VGDF + BC : We adapt VGDF to the Offline-Online setting by simply integrating the behavior cloning loss following (10). The training is repeated for 10 times per step with the target domain the same as the baseline methods. For the trade-off between the policy gradient and behavior cloning, we use the value-normalized regularization following the TD3 + BC [22] work and set the constant α as 5 . Furthermore, we remove the exploration policy proposed in Section 5.1 since the online access to the source domain is no longer available in the offline-online setting.

F Additional Experiment Results

F.1 Quantifying Dynamics Shifts via FVP

In this section, we investigate whether the estimation of the value differences can quantify the difference across domains. Specifically, in different target domains of the same source domain, we demonstrate the estimation of FVP in two target domains. As the results show in Figure 11, the FVP differs in environments with different dynamics shifts (Kinematic or morphology). We observe that the FVP values in two target domains gradually approach each other in three out of four environments (HalfCheetah, Walker, Hopper), while the values in Ant remain relatively stationary. Furthermore, the FVP values in target domains with kinematic shifts are lower than those with morphology shifts across all four environments, which could result from the mismatched state space due to the limited joint ranges of robots in the target domain. Given the differences across different environments, we believe the FVP estimation could be used to quantify the domain differences.

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0.0

0.1

0.2

0.3

0.4

Approximated FVP

[

]

HalfCheetah

Kinematic

Morphology

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

Ant

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

Walker

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

Hopper

F.2 Sensitivity to Ensemble Size

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

5000

Return

HalfCheetah - broken back thigh

M = 7

M = 5

M = 3

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

Return

HalfCheetah - no thighs

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

Return

Hopper - broken joints

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

Return

Hopper - big head

We have introduced the dynamics model ensemble to capture the epistemic uncertainty induced by the limited samples from the target domain. However, training the ensemble of the dynamics model takes extra computation resources. Unlike prior works in model-based RL [30, 61] that utilize the generated samples for training, we measure the value difference with the help of the generated samples. Therefore, we aim to investigate whether a smaller ensemble size is sufficient to achieve competitive asymptotic performance. Here we set the ensemble size as different values ( M = 7 in the original implementation) and run experiments in four environments. As the results show in Figure 12, variants with a small ensemble size ( e.g. , M = 3 or M = 5 ) can achieve identical asymptotic performance compared to the variant with a large ensemble size ( e.g. , M = 7 ) in three out of four environments.

F.3 What about Importance Weighting via FVP instead of Rejection Sampling?

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

5000

Return

HalfCheetah - broken back thigh

Rejection Sampling

Steps in Target Domain (×10 5 )

500

1000

1500

2000

2500

3000

3500

Return

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

3500

Return

Walker - broken right foot

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

Return

Hopper - broken joints

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

1000

2000

3000

Return

HalfCheetah - no thighs

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

500

1000

1500

2000

2500

3000

Return

Ant - short feet

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

Return

Walker - no right thigh

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

Return

Hopper - big head

In the case of data selection based on the estimated FVP (fictitious value proximity in Eq. (6), one may wonder about using importance weighting via the FVP rather than rejection sampling, which might be sample-inefficient due to the discarded partial data. Here we implement a variant of our algorithm that performs importance weighting with the estimated fictitious value proximity. Specifically, we train the value functions following:

We compare the variant with the original algorithm using rejection sampling in all eight environments and demonstrate the results in Figure 13. The original algorithm using rejection sampling outperforms the variant with importance weighting in almost all environments. The accuracy of the value proximity depends on the generated state and the value function. Thus, the estimation of FVP could be biased due to the inaccurate dynamics models and value functions in the early training stage, in which case naively utilizing the source domain samples weighted by the FVP can harm the policy performance concerning the target domain. In contrast, rejection sampling that only utilizes a small portion of source domain samples alleviates the negative effect of the source domain samples.

F.4 What about Data Filtering via Value instead of FVP?

Prior works have examined sharing data across tasks with different reward functions rather than dynamics [75]. To investigate whether selectively sharing data with a high Q value can address the

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

5000

Return

HalfCheetah - broken back thigh

Filtering via FVP

Filtering via Value

Steps in Target Domain (×10 5 )

500

1000

1500

2000

2500

3000

3500

Return

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

3500

Return

Walker - broken right foot

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

Return

Hopper - broken joints

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

1000

2000

3000

Return

HalfCheetah - no thighs

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

500

1000

1500

2000

2500

3000

3500

Return

Ant - short feet

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

Return

Walker - no right thigh

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

Return

Hopper - big head

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

3500

Return

Walker - broken right foot

VGDF

DGDF

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

1000

2000

3000

4000

Return

Walker - no right thigh

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

Return

Hopper - broken joints

0.0

0.2

0.4

0.6

0.8

1.0

Steps in Target Domain (×10 5 )

0

500

1000

1500

2000

2500

3000

Return

Hopper - big head

online dynamics adaptation problem, we propose a variant of our algorithm that shares partial data with a relatively high Q value from the source domain. Specifically, we train the value functions following:

where Q ξ % is the top ξ -quantile Q value of a batch of source domain samples. We set ξ % as 25% , the same as our implementation. We compare the variant with the original algorithm in all eight environments and demonstrate the results in Figure 14. The results demonstrate that the original algorithm outperforms the variant using data filtering via value in four of eight environments. Due to the dynamics mismatch, a state-action pair from the source domain will lead to inconsistent states concerning two domains. Therefore, directly utilizing the transitions with high Q value without considering the consistency of the next state would provide a counterfactual value target for the state-action pair, which can result in an improper value estimation for learning.

F.5 Comparison with Dynamics-guided Data Filtering

To investigate the effect of value consistency, we perform the ablation study by comparing VGDF to a variant that shares partial data based on dynamics discrepancies, i.e. , Dynamics-guided Data Filtering (DGDF). Specifically, we estimate the dynamics discrepancy via the learned classifiers

following the prior works [17, 49]. Same as VGDF, we share the source domain transitions whose estimated dynamics difference is smaller than the quantile value. We set the selection ratios as 25% , the same as our implementation. The results demonstrate that the original algorithm outperforms the variant in three out of four environments, validating the superior effect of the value consistency compared to the dynamics discrepancy.

F.6 Extended Results in Pybullet Environments

To investigate the generality of VGDF, we perform additional experiments in PyBullet-HalfCheetah and PyBullet-Hopper from PyBullet environments [12] which utilize Bullet as the physical engine instead of Mujoco. We first provide the details of the dynamics gap in the environments. In both environments, we regard the original environments as the source domains. In PyBullet-Hopper, we devised the target domain by increasing the torso size from 0.05 to 0.15, to simulate the morphology change. In PyBullet-HalfCheetah, we constrain the joint range of the front thigh from [ -1 . 5 , 0 . 8] to [ -1 . 5 , 0 . 4] , and the joint range of the front shin from [ -1 . 2 , 1 . 1] to [ -1 . 2 , 0 . 1] , to simulate the broken joint scenario that is widely used in related works.

The results are shown in the Table 3, and we report the performance of all algorithms concerning different numbers of target domain samples. All results are averaged across five runs with different seeds. The results demonstrate that VGDF consistently outperforms baselines given different number of target domain data, demonstrating the generality of our method.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method that does domain adaptation for RL. Unlike other approaches that do this via domain randomization, offline interaction from the target env, or adjusting the sim with system ID from a small real dataset, the aim is to learn from a large source of sim transitions and a small number of **online** interactions.\n\nThe closest comparison to the work is DARC, which was also evaluated in a similar setting. DARC is from a class of methods that learns a reward correction term $\\Delta r$ that captures the change in environment dynamics between source and target domain, and estimates $\\Delta r$ using classifiers. This work suggests that DARC's estimation method is too conservative. In particular, given two tuples $(s,a,s_1')$ and $(s,a,s_2')$, if $V^\\pi_{tar}(s_1') \\approx V^\\pi_{tar}(s_2')$, then for the purposes of value estimation $s_1'$ and $s_2'$ can be considered similar, even if the individual states themselves are quite different. This is an example of a case where DARC could produce a larger reward penalty.\n\nThis work aims to estimate $V^\\pi_{tar}(s)$ from a small number of interactions in the target domain + many interactions in the source domain. To do so, an ensemble of dynamics models $T_\\phi(s'|s,a)$ is learned from target data. Sampling target state $s_{tar}'$ from the $T_\\phi(s'|s,a)$ family gives an ensemble of Q-value estimates $Q_{tar}(s,a)$, and this lets us model $V^\\pi_{tar}(s)$ as a random variable with some amount of uncertainty. The function is then updated with all target data, and some source data, where the source data is used only if it is measured as sufficiently \"in-distribution\" to the target environment. (In practice the authors use the top 25% most \"in-distribution\" source data.)\n\nSTRENGTHS:\nI found the theoretical arguments reasonably clear, modulo a few typos (see later section), and on double-checking them, everything looks correct.\n\nI think the core idea makes sense: that for the purposes of learning, if you assume your value estimate $V$ is accurate enough, then it is sufficient to compare environments based on your value estimate $V(s_{src})$ vs $V(s_{tar})$ since the value function encapsulates all future variation. This idea is compelling.\n\nWEAKNESSES:\nIn terms of references, there are some approaches that use value function equivalence as an objective for learning their manipulable sim (i.e. RL-CycleGAN, if you view learning an image-to-image GAN as a manipulable sim).\n\nMy more specific reservation is with the comparison to DARC, the most competitive of the baselines. The GridWorld example given shows an environment where DARC is overly pessimistic in bottleneck states, and fails to explore outside of that region. DARC has theoretical guarantees of correctness, so what is going on here? Perhaps in practice, the theory is too conservative (the field has seen similar things with TRPO / PPO using less-conservative approximations of theory).\n\nIn this scenario, my natural response would be to not to derive a new method, but to try weighting the reward correction term $\\Delta r$ in DARC by some hyperparameter. It is not very theoretically principled but would probably handle the pessimism alright. Does this approach meaningfully improve on DARC if the conservativism term was weighted? It's unclear.\n\nSetting this aside for now, if $V$ is learned well, then this paper (VGDF) makes sense. But to me this seems like a very strong \"if\" to make. It's a little chicken and egg, where $V$ should only be learned well if we did not need the source data in the first place? To determine the closest section of source data, we must already learn a 1-step dynamics model of the target environment $T(s_{t+1}|s,a)$, and estimate $V_{tar}(s')$ to estimate the future value from $s'$. If both those pre-requisites are true, it feels like we ought to just use a model-based method instead.\n\nAs implemented, the paper always uses the 25% closest segment of the source data, but there is no guarantee this 25% closest segment is actually that close to the target data. The ablations in Figure 6 suggest that the separation function is doing something, but I am a bit concerned that performance does not seem to differ much for many different thresholds $\\xi$.\n\nI am also not so fond of the exploration policy $\\pi_E$. It makes sense but I believe the baselines do not use a separately trained exploration policy (please correct me on this if wrong). It doesn't seem like $\\pi_E$ is a major component based on the ablation run, so this is a lesser concern for me.\n\nQUESTIONS:\nAre there any results suggesting the accuracy of the value function $V$ during training? Or at least suggesting that any inaccuracies in $V$ are correlated during training?\n\nI would also appreciate more details on how the Gaussian dynamics models $T_\\phi$ were learned - given that the paper's central argument is to measure state distance by $V_{tar}(s)$, the method section for how $V_{tar}$ is learned seems oddly rushed.\n\n\nMisc questions:\n\nline 580 of appendix: it should be $W_{j+1} = ... + V(s_{j+1})$ instead of $V(s_j)$ right?\n\nline 565 of appendix: I'm not sure why the proof is 4 lines. Lemma C.1 gives $\\eta_1(\\pi) - \\eta_2(\\pi) = \\frac{\\gamma}{1-\\gamma} E[ E_{P_1}[V_{tar}(s')] - E_{P_2}[V_{tar}(s')] ]$. The end of Theorem B.2 is equivalent to, $\\eta_1(\\pi) - \\eta_2(\\pi) \\le$ \"the outer expectation, if you take the absolute value of the insides before integrating\". So I'm not sure why there's a step where the $P_{src}$ and $P_{tar}$ terms are shuffled around before taking the absolute value, seems like needless notation unless I'm missing something.\n\nLIMITATIONS:\nLimitations seems addressed fine.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is concerned with the online dynamics adaptation problem, where an agent is tasked to generalize from a source domain with cheap access to a target domain with a limited online interaction budget. The approach filters what data from the source domain is used in the target domain based on whether the state-action pair will result in a state of similar value in both source and target domain. This approach is theoretically justified and motivated in a toy example. The authors show empirical results in modified versions of the Deepmind control suite.\n\nSTRENGTHS:\nThe paper benefits from a relatively tight relation between theoretical motivation and practical implementation. The \"value discrepancy perspective\" is original in this context, and the authors carefully compare and ablate their method. The paper is easy to read and to understand. As sim2real transfer is a very real and pressing issue in robotics and control, I think the paper makes a significant contribution.\n\nWEAKNESSES:\nThe paper could potentially benefit from additional experiments in environments that are not in the \"short-term control\" regime that is typical for the Deepmind control suite, but more long-horizion like physical manipulation scenarios or a more complex version of the maze that is used for the motivating example. Since in such scenarios, value assignment is generally harder to do correctly, it might be interesting to see how the performance of value-based data sharing is affected in such a case.\n\nQUESTIONS:\n- I did not really understand why the fact that DARC learns overly pessimistic value estimations should be (as argued in section 4.1 and elsewhere) a direct consequence of DARC being dynamics-based rather than value-based? Would it not be possible to create a \"more forgiving\" version of DARC that, for example, uses a data-filtering mechanism rather than the (arguably over-pessimistic) reward shaping mechanism it currently uses? If I did not misunderstand the motivating example, I feel like it does not really disentangle the two aspects of (1) reward shaping vs. data filtering and (2) dynamics-based vs. value-based, but then the difference in results is mostly assigned to (2). The argument that value difference is more long-horizon than dynamics difference makes sense to me, though.\n\nLIMITATIONS:\nThe limitations section is quite brief. I appreciate that space is very constrained, but maybe it would improve the paper to expand this a little for the camera-ready version.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a setting of online dynamics adaptation, where the goal is to train a near-optimal policy in the target domain using transition data from the source domain and the target domain with different dynamics. The authors propose to select source domain data to train Q-functions if the value discrepancy between the source and the target domain is minimal. The authors propose Fictitious Value Proximity (FVP) that represents the likelihood of the source domain state value and select source domain transitions with the top 25% quantile of FVP. The authors evaluated the proposed method on environments with different dynamics, including kinematics and morphology change.\n\nSTRENGTHS:\n1. This paper tackles the problem of generalizing policies across dynamics mismatch, which is significant in reinforcement learning.\n\n2. The writing and the clarity are good. Also, this paper includes a theoretical performance bound controlled by value difference to support the claim.\n\nWEAKNESSES:\n1. The proposed idea seems to be simple and obvious. VGDF wants to select source domain data with similar state transition dynamics for additional training data, and it uses a value function to measure the similarity of state transition dynamics.\n\n2. Although VGDF demonstrates superior performance baselines in some environments, it has comparable or inferior results to a fine-tuning method on the target domain in other environments.\n\nQUESTIONS:\n1. There seems to be a discrepancy between the method described in the paper and the code implementation of it. In the paper, VGDF selects the source domain samples based on the likelihood of the next state value $V(s’)$. On the other hand, in the code, VGDF selects the source domain samples based on the likelihood of $r + \\gamma V(s’)$, which approximates the current state value $V(s)$. In the code, the ensemble of dynamics models predicts the reward $r$ and the next state $s’$ in the target domain (line 286 in vgdf.py). These predictions are then used to compute Fictitious Value Ensemble (FVE) of $r + \\gamma V(s’)$ (lines 244 and 303 in vgdf.py) and Fictitious Value Proximity (FVP) of $r + \\gamma V(s’)$ on source domain samples. (line 261 in vgdf.py). This means that VGDF chooses source domain samples based on $V(s)$ instead of $V(s’)$, which contradicts the description in the paper. It is unclear why the authors chose to use $r + \\gamma V(s’)$ instead of $V(s’)$ in the code implementation.\n\n2. According to Fig. 7, using $\\pi_E$ seems insignificant. How exactly the exploration policy behaves in the source domain? I wonder whether maximizing $Q_{UB} = \\max\\{Q1, Q2\\}$ actually leads to optimistic exploration.\n\nLIMITATIONS:\nThe authors included the limitations in the Conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDisclaimer: I found it challenging to fully comprehend the paper. This may be due to either inadequate clarity in the writing or my own limitations in understanding the subject matter.\n\nThis work introduces a method called Value-Guided Data Filtering (VGDF), which aims to enable online dynamic adaptation. By utilizing a set of state-action pairs from a source domain, VGDF selects relevant transitions to train a policy that performs well in a target domain with differing dynamics, all while not requiring the need for extensive interactions with the target domain. To assess the effectiveness of the proposed method, experiments are conducted on four Gym Mujoco environments that feature diverse, dynamic shifts, such as kinematic changes and morphological variations.\n\nSTRENGTHS:\n- The main results depicted in Figure 4 exhibit considerable strength and indicate that VGDF outperforms the baseline methods.\n- The proposed method is supported by a theoretical analysis presented in Section 4.\n- The authors have conducted a good number of analyses, including ablation studies and quantification of dynamic mismatch.\n\nWEAKNESSES:\nThe main weakness of this paper lies in its lack of clarity in communication and presentation. Overall, I found it challenging to follow the arguments and ideas presented. Furthermore, it remains unclear what the significant contributions of this work are and the novel aspects of the proposed filtering mechanism. For instance, in lines 52-58, the authors attempt to summarize their contributions. However, empirically demonstrating the superiority of their method (contribution number 4) is not considered a contribution itself, as every claim requires proper evaluation.\n\nAnother weakness of this work is the limited breadth of the experimental analysis. While the main results in Figure 4 undoubtedly showcase excellent performance for VGDF in Gym Mujoco environments, it is uncertain whether this generalizes to other environments beyond that scope.\n\nQUESTIONS:\nI strongly recommend that the authors undertake a significant revision of their paper, particularly focusing on Section 4 and Section 5. Additionally, it is important to ensure that the paper is self-contained and does not necessitate reading the appendix. Therefore, I suggest that the authors incorporate additional details regarding the experimental setup from the appendix into the main body of the paper.\n\nLIMITATIONS:\nThe authors seem to have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the online dynamics adaptation problem where the agent has access to a large number of samples or an offline dataset from a source domain, and must adapt with a smaller number of samples in a target domain.\n\nTo address this problem, this paper introduces a framework for using a value discrepancy perspective. Specifically, they consider transitions with consistent value targets equivalent, even if they have different dynamics. This is different from prior works that focus purely on dynamics discrepancy, and encourage the agent to avoid state-actions with different dynamics. Towards this end, they show some theoretical analysis demonstrating a bound on policy performance given value consistency.\n\nGiven this perspective, they present the Value-Guided Data Filtering (VGDF) algorithm which uses value consistency to do selective data sharing from the source domain to the target domain.\n\nThe practical implementation of this algorithm involves learning an ensemble of gaussian dynamics models for the target domain. These different models are used to generate different fictitious transitions that can be used to estimate a gaussian distribution of potential values in the target domain for a given state-action pair from the source domain. Thus, the consistency of state-actions from the source domain are evaluated by Fictitious Value Proximity (FVP), which computes the likelihood of the estimated value of the source next state given the estimated gaussian distribution of potential values in the target domain. \n\nThey train a Q function for the target domain by using the samples from the target domain and samples from the source domain if they are in the top $\\xi$-quantile likelihood estimation of the sampled minibatch. They train an evaluation policy and exploration policy with SAC, with the main distinction being the exploration policy is trained to optimized the max of the 2 Q functions instead of the minimum. To tackle the setting where the agent only has an offline dataset collected in the source domain, they introduce a BC term to their optimization similar to TD3 + BC.\n\nThey evaluate their results in several Mujoco environments where the target domain introduces a kinematic shift or morphological shift to the robot. They generally find that their method outperforms their baselines in both the online and offline setting. The ablations demonstrate the benefits of the filtering by FVP and training the separate exploration policy. Finally, they demonstrate how FVP can be used to analyze different types of dynamics shifts in the target domain.\n\nSTRENGTHS:\nThe paper is well written, clear, and flows well. I believe an expert could reproduce the algorithm given the details in the paper.\n\nI believe the discussion on the value discrepancy perspective is an interesting contribution to the field that would interest other researchers.  I think in particular the theoretical results in section 4.2 and the motivating examples and figures in section 4.1 do a great job demonstrating the validity of this perspective.\n\nAs far as I am aware, VGDF is a novel and interesting algorithm that is well explained and justified by their value discrepancy perspective. The authors include good experimental results and ablations that demonstrate the effectiveness of their algorithm and specific design choices.\n\nAdditionally, I think the setting of doing online dynamics adaption from an offline dataset is a currently underexplored topic that should get more interest from the research community. The VGDF + BC is a seemingly simple yet effective extension of VGDF that seems to do well in this setting.\n\nWEAKNESSES:\nIn my opinion, there are no major weaknesses in this paper, but I do have 2 small criticisms I would like to be addressed.\n\nI would appreciate comparisons in Section 6.3 to more offline RL algorithms that were designed for offline pretaining and online finetuning, like IQL.\n\nI believe there are many real-world scenarios where online samples are prohibitively expensive compared to simulated samples or previously collected offline data. Thus, I would appreciate more ablations with higher numbers for $\\Gamma$, but less online samples.\n\nQUESTIONS:\nDo you think your method will be effective in doing sim-to-real transfer? It seems to be a motivating example, but their are no results that directly indicate that your method will be better at sim-to-real transfer, than prior approaches.\n\nLIMITATIONS:\nNo obvious limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}