Cross-Domain Policy Adaptation via Value-Guided Data Filtering
Authors: Kang Xu
Chenjia Bai
Xiaoteng Ma
Dong Wang
Bin Zhao
Zhen Wang
Xuelong Li
Wei Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning; Domain Adaptation; Online Dynamics Adaptation

Abstract: Generalizing policies across different domains with dynamics mismatch poses a significant challenge in reinforcement learning. For example, a robot learns the policy in a simulator, but when it is deployed in the real world, the dynamics of the environment may be different. Given the source and target domain with dynamics mismatch, we consider the online dynamics adaptation problem, in which case the agent can access sufficient source domain data while online interactions with the target domain are limited. Existing research has attempted to solve the problem from the dynamics discrepancy perspective. In this work, we reveal the limitations of these methods and explore the problem from the value difference perspective via a novel insight on the value consistency across domains. Specifically, we present the Value-Guided Data Filtering (VGDF) algorithm, which selectively shares transitions from the source domain based on the proximity of paired value targets across the two domains. Empirical results on various environments with kinematic and morphology shifts demonstrate that our method achieves superior performance compared to prior approaches.

Cross-Domain Policy Adaptation via Value-Guided Data Filtering
Kang Xu 2+
Chenjia Bai
Xiaoteng Ma²
Dong Wang*
Bin Zhao?
Zhen Wang*
Xuelong Lif
Wei Lift
Fudan University
Shanghai Artificial Intelligence Laboratory
3 Tsinghua University
Northwestern Polytechnical University
Abstract
Generalizing policies across different domains with dynamics mismatch poses a significant challenge in reinforcement learning. For example, robot learns the policy in simulator, but when deployed in the real world, the dynamics of the environment may be different. Given the source and target domain with dynamics mismatch, we consider the online dynamics adaptation problem, in which case the agent can access sufficient source domain data while online interactions with the target domain are limited. Existing research has attempted to solve the problem from the dynamics discrepancy perspective. In this work. reveal the limitations these methods and explore the problem from the value difference perspective via novel insight the value consistency across domains. Specifically, we present the Value-Guided Data Filtering (VGDF) algorithm, which selectively shares transitions from the source domain based on the proximity of paired value targets across the two domains. Empirical results on various environments with kinematic and morphology shifts demonstrate that our method achieves superior performance compared to prior approaches.
Introduction
Reinforcement Learning (RL) has demonstrated the ability to train highly effective policies with complex behaviors through extensive interactions with the environment [62, 59, 2]. However, in many situations, extensive interactions are infeasible due to the data collection costs and the potential safety hazards associated with domains such roboties [33] and medical treatments [54]. To address the issue, one approach is to interact with a surrogate environment, such as a simulator, and then transfer the learned policy to the original domain. However, unbiased simulator may be unavailable due to the complex system dynamics or unexpected disturbances in the target scenario, leading to dynamics mismatch. Such mismatch crucial for the sim-to-real problem in robotics [1, 38, 51] and may cause performance degradation of the learned policy in the target domain. In this work, we focus on the dynamics adaptation problem, where we aim train well- well-performing policy for the target domain, given the source domain with the dynamics mismatch.
Recent research has tackled the adaptation over dynamics mismatch through various techniques, such as domain randomization [56, 53, 45], system identification [77], or simulator calibration [8], that require domain knowledge privileged access the physical system. Other methods have explored
Part of this work was done during Kang Xu's internship at Shanghai Artificial Intelligence Laboratory. "oorreppondecce to Chenjia Bai <baichenjia @ pilab.org.cn>, Wei Li <fd lillliieeee fudan.edu.cn>.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Daca
Trin
Source Bonaa'
Train
RL
Calbration
Targe Domzin
Torgst
Terget
Deplosment
Daman
Deployment
Doman
Deplesment
Deployment
(6) Dynamica Runcomiaatioon System Leirr Heatton//Woaa
Jal Dynamics es Adth Samoosiiann/OOlll Cuteest
Ad Dynamis Adaptation with Maripulable Semulater
14 Online Dyromiss Adaptation
Figure 1: Semantic illustration of main settings for dynamics adaptation problem. Methods in the first three categories require different assumptions, such as a wide range of source domains, demonstrations from the target domain, or a manipulable simulator. We focus on more general setting, online dynamics adaptation, only requiring limited online interactions with the target domain.
the adaptation problem in specific scenarios, such as those with expert demonstrations [41, 32] or offline datasets 49], while the effectiveness of these methods heavily depends on the optimality of demonstrations or the quality of the datasets. In contrast to these works, we consider a more general setting called online dynamics adaptation, where the agent can access sufficient source domain data and a limited number of online interactions with the target domain. We compare the settings for the dynamics adaptation problem in Figure 1.
To address the online dynamics adaptation problem, prior works mainly focus on the single-step dynamics discrepancy and practically eliminating the gap via different ways [17, 14]. However, we empirically demonstrate the limitation of the methods through a motivation example, suggesting their effectiveness heavily relies strong assumptions about the transferability of paired domains. Theoretically, we formulate the performance bound of the learned policy with respect to the dynamics discrepancy term, which provides an explicit interpretation of the results. To address the problem, we focus on the value discrepancy between paired transitions across domains, motivated by the key idea: the transitions with consistent value largets can be seen equivalent for policy adaptation. Based on the insight, we proposed a simple yet efficient algorithm called Value-Guided Data Filtering (VGDF) for online dynamics adaptation via selective data sharing. Specifically, we use a learned target domain dynamics model to obtain paired transitions based on the source domain state-action pair. The transitions are shared from the source to the target domain only if the value targets of the imagined target domain transition and that of the source domain transition are close. Compared to previous methods that utilize the single-step dynamics gap, our method measures value discrepancies to capture long-term differences between two domains for better adaptation.
Our contributions be summarized follows: 1) We reveal the limitations of prior dynamics-based based methods and propose the value discrepancy perspective with theoretical analysis. 2) To provide a practical instantiation, we propose VGDF for online dynamics adaptation via selective data sharing. 3) We extend VGDF to a more practical setting with an offline source domain dataset and propose a variant algorithm motivated by novel theoretical results. 4) We empirically demonstrate the superior performance of our method given significant dynamics shifts, including kinematics and morphology mismatch, compared to previous methods.
2 Related Work
Domain adaptation in RL. Different from domain adaptation in supervised learning where dif- ferent domains correspond to distinct data distributions [34], different domains in RL. can differ in observation space [26], transition dynamics [56, 77, 17], embodiment [79, 43), or reward func- tions [16, 81, 57]. In this work, we focus on domain adaptation with dynamics discrepancies. Prior works utilizing meta RL [76, 48, 55], domain randomization [56, 53, 45], and system identifica- tion [80, 77, 15, 74] all assume the access the distribution of training environments and rely on the hypothesis that the source and target domains are drawn from the same distribution. Another line of work has proposed to handle domain adaptation given expert demonstrations from the target domain [41, 32, 27]. These approaches align the state visitation distributions of the trained policy in the source domain to the distribution of the expert demonstrations in the target domain through state-action correspondences [79] or imitation learning [28, 21, 72]. However, near-optimal demon- strations can be challenging to acquire in some tasks. More recent works have explored the dynamics adaptation given an offline dataset collected in the target domain [42, 49], while the performance of

---

the trained policy depends on the quality of the dataset [50]. Orthogonal to these settings, we focus on a general paradigm where a relatively small number of online interactions with the target domain are accessible.
Online dynamics adaptation. Given limited online interactions with the target domain, several works calibrate the dynamics of the source domain by adjusting the physical parameters of the simulator [8, 58, 15, 47J, while they assume the access of manipulable simulator. Action transformation methods correct the transitions collected in the source domain by learning dynamics models of the two domains [25, 14. 78]. However, the learned model can be inaccurate, which results in model exploitation and performance degradation [30, 31]. Furthermore, the work that compensates the dynamics gap by modifying the reward function [17] is practical only if the policy that performs well in both domains exists. Instead. we do not assume the dynamics-agnosse policy exists and demonstrate the effectiveness of our method when such an assumption does not hold.
Knowledge transfer in RI. Knowledge transfer has been proposed to reuse the knowledge from other tasks to boost the training for the current task [69, 37j. The transferred knowledge can be modules (e.g., policy) [52, 9, 4], representations [5], and experiences [29, 39, 75, 68]. Our method is related to works transferring experiences. However, prior works focus on transferring between tasks with different reward functions instead of dynamics. When the dynamics changes, the direct adoption of commonly used temporal difference error [63] or advantage function [60] in previous works [29, 39, 68] would be inappropriate due to the shifted transition probabilities across domains. In contrast, we introduce novel measurements to evaluate the usefulness of the source domain transitions to tackle the dynamics shift problem specifically.
Theories on learning with dynamics mismatch. The performance guarantee of policy trained with imaginary transitions from an inaccurate dynamics model has been analyzed in prior Dyna- style [64, 65, 67] model-based RL algorithms [44, 30, 61]. The theoretical results inspire us to formulate performance guarantees in the context of dynamics adaptation.
3 Preliminaries and Problem Statement
We consider two infinite-horizon Markov Decision Processes (MDP) Marc := (S.A, P 7,7, Po) and Mtar == A,, Ptar, T, T/ Po) for the source domain and the target domain, respectively. The two domains share the same state space S, action space A, reward function r: S x A + R with range [0, Tmaa]] discount factor 7 E [0, 1), and the initial state distribution pa : : + [0, 1]. The two domains differ on the transition probabilities, i.e., (s (s, a) and (s)sss)),))
We define the probability that a policy π encounters state P (s) We denote the normalized probability that a policy π encounters state in M as (s) == (1 7) (PPM.(s) and the normalized probability that a policy encounters state-action pair (s, a) in M is pM (s, a): = (1 7) Ei (s) The performance of a policy x in M as is formally defined as (T) == Es,arp% (r(s. a)].
the time step in MDP M as
We focus on the online dynamics adaptation problem where limited online interactions with the target domain are accessible, which can be defined as follows:
Definition 3.1. (Online Dynamics Adaptation) Given source domain Mare and target domain
Mear with different dynamics, assume sufficient data from the source domain (online offline)
and a relatively small number of online interactions with Mior (e.g., :== # source domein data # target domain data 10),
hoping obtain near-optimal policy π concerning the target domain M
The prior work [17] also focuses on the online dynamics adaptation problem with online source domain interactions. The proposed algorithm DARC estimates the dynamics discrepancy via learned domain classifiers and further introduces reward correction (i.e., Ar(s, a, 8') x log (Prar(s')ss (s((s,))) (s'ss, |s, a))) to optimize policy together with the task reward (i.e., r(s, a) Ar(s, a, s')), discouraging the agent from dynomics-inconsistent behaviors in the source domain.
4 Guaranteeing Policy Performance from a Value Discrepancy Perspective
In this section, we will first present an example demonstrating the limitation of the prior method considering the dynamics discrepancy. Following that, we provide a theoretical analysis of the
3

---

State Visitation
Learned Table
Source
Domain
Source Domain
Target
Domain
Target
Ours
DARC
Ours
DARC
(a)
(b)
(c)
Figure 2: The illustrations and results of the motivation experiment. (a) Illustration of the source and target domains the grid world environment. The red dot and green square represent the agent and goal, respectively. (b) Visualization of the state visitation in both domains. 'hhe darker color suggests higher visitation probabilities. Our method guides the agent to reach regions with high target domain values while the agent trained by DARC is stuck in the room. (c) Visualization of the learned Q tables. Four triangles represent four actions; the darker color suggests higher value estimation. Our method learns the optimal Q table whose greedy policy leads the agent the goal of the target domain, while DARC fails due to pessimistic values the crucial state-action pairs with dynamics mismatch.
dynamics- based method to provide an interpretation of the experiment results. Finally, we introduce novel perspective on value discrepancies across domains for the online dynamics adaptation problem.
4.1 Motivation Example
We start with a 2D grid world task shown Figure (a), where the agent represented by the dot needs to navigate to the green square representing the goal. We design source and target domains with different layouts and train a policy to reach the goal successfully in the target domain. We investigate the performance of DARC [17] that trains the policy with Synamics-guider reward correction and our proposed method (Section 5), using tabular Q-learning [73] as the backbone for all methods. Detailed environment settings shown in Appendix D.
As the empirical state visitations and the learned Q tables show in Figure 2, DARC stuck in the room and fails to obtain near-optimal Q-values, leading to poor performance. Specifically, we circle out four positions where specific actions will lead to the states with a dynamics mismatch concerning the two domains. Due to the introduced reward correction on the source domain transitions with dynamics mismatch, DARC learns overly pessimistic value estimations pairs, which hinders the agent from the optimal trajectory concerning the target domain. However, the values of the following inconsistent states, induced by the particular state-action pairs, are not significantly different concerning the target domain. The value difference quantifies the discrepancy of the long-term behaviors rather than single-step dynamics. Motivated by the value discrepancy perspective, proposed method (Section 5.1) demonstrates superior performance.
particular state-action
4.2 Theoretical Interpretations and Value Discrepancy Perspective
To provide rigorous interpretations for the results, we derive a performance guarantee for the dynamics- guided methods, which mainly build the theories proposed in prior methods [30, 17].
Theorem 4.1. (Performance bound controlled by dynamics discrepancy.) Denote the source domain and target domain with different dynamics as Myre and Maar, respectively. We have the performance difference of any policy evaluated under Myrc and Miar be bounded below,
2-yTmax
MMM (T) 2 TIMor (TT)
EE (Drv (Psrel Perre(11a (-18,, a) Pear Ptar(113. (-18, (-),
7)
(a) dynamics discrepancy
The proof of Theorem 4.1 is given in Appendix B. We observe that the derived performance bound in (1) is controlled by the dynamics discrepancy term (a). Intuitively, the performance difference would
be minor when the dynamics discrepancy between the two domains is negligible. DARC [17] applies

---

the Pinsker's inequality [13] and derives the following form:
1]Maar (T) 2 1/Mrro (T) (1 (1-7) 7)2
Vmax
VIE Pore
[DEL (Pore(tss;)) a) Piar (-1s, a))
= -lMarr (T) + (1 7) 2Ep [log (Ptar 1/////1/1/ 8, a) a Perc (s')s,
Vmmax
(2)
Based on the result in (2), DARC optimizes the policy by converting the second term in RHS to a reward correction (i.e., Ar based adaptation. However, given the transition from the source domain (i.e., Pare(s'le (s'|8 18, a) R 1), the reward correction will lead to significant penalty (i.e. (og(Ptar(s')) (s'((,))))))))))))) s,a)/ likelihood estimation of the transition concerning the target domain is low (i.e., Peor (s' (s'|s, a) x 0). Consequently, the value estimation of the transition with dynamics mismatch tends to be overly pessimistic as shown in Figure 2 (c), which hinders learning an effective policy concerning the target domain.
(ota(Parr a)/Psrdd a)))), leading the dynamics discrepancy-
a Ps (re(ssll (,))))))))))))))))))))
0) if the
Instead of myopically considering the single-step dynamics mismatch, we claim that the transitions with significant dynamics mismatch can be equivalent concerning the value estimations that evaluate the long-term behaviors. Due to the dynamics shift across domains, a state- action pair (i.e. (s, a)) would lead to different next-states (i.e., Kavez &tar), the paired transitions are nearly equivalent for temporal different learning if the induced value estimations are close (i.e., [V(s'sre) (s(.ar) S 6). Motivated by this, derive performance guarantee from the value difference perspective.
Theorem 4.2. (Performance bound controlled by value difference.) Denote source domain and target domain Mare and Mear, respectively. We have the performance guarantee of any policy over the two MDPs:
Ep.rc [VMtrr (s')) Eear [V (
T.Mar (④) 2 aMrre (T)
Ep
(3)
(al: value difference
The proof of Theorem 4.2 is given in Appendix B. The value difference term provides novel perspective: the performance can be guaranteed if the transitions from the source domain lead to consistent value targets in the target domain. The result further highlights the value consistency perspective for the online dynamics adaptation problem.
5 Value-Guided Data Filtering
In this section, propose Value-Guided Data Filtering (VGDF), a simple yet efficient algorithm for online domain adaptation via selective data sharing- Then we introduce the setting with offline source domain data and variant of VGDF based novel theoretical results. The pseudocodes shown in Appendix A, and the illustration of VGDF is shown in Figure 3.
5.1 Dynamics Adaptation by Seleetive Data Sharing
Inspired by the performance bound proposed in Theorem 4.2, we can guarantee the policy performance by controlling the value difference term in (3). As discussed in Section 4.2, the paired transitions concerning two domains, induced by the same state-action pair, can be regarded as equivalent for temporal difference learning when the corresponding values are close. Thus, we propose to select source domain transitions with minor value discrepancies for dynamics adaptation.
To select rational transitions from the source domain, we need compare the value differences of paired transitions based on the same source domain state-action pair (Sare: aare). ). Formally, given a vate-action pair (Sare; asre) from the source domain, our objective is to estimate whether the value-difference between star and s is sufficiently small, i.e.,
A(Sarey Uarc) 1 (IV (star) Valra (S'rc)) <)).
where s'yar ~ Ptor( (. Sares aare): d'rr ~ Paral (: Sares asre). 1 denotes the indicator function and e be predefined threshold.
To obtain A (src), we need to perform policy evaluation over the states to obtain the value estimations given the paired next states (i.e., s'res sor), formulated in Eq. (4), Monte Carlo (MC)
5

---

| #1 vadess..)1
Dire
Dere
(6.0)
Date Fitering
(5.0,
Uo
(IL))
Pear
RL
Rejetiin
1(A Age
Figure 3: Semantic illustration of VGDE. We tackle online dynamics adaptation by selectively sharing the source domain data, and the RI. denotes any off-the-shelf off-policy RL algorithm.
evaluation can provide unbiased values by rolling the policy starting from specific states [66]. However, since the environment is not manipulable, we cannot perform MC evaluation from arbitrary states. Thus, we propose to use an estimated value function for policy evaluation. In this work, we adopt the Fitted Q Evaluation (FQE) [46] that is widely used in off oflliol RL algorithms [40, 23, 24).
Specifically, we utilize a learned Q function Qa S x A + I for evaluation.
problem is that the corresponding target domain (Ssrc: (are) is unavailable in practice. To achieve this, we train a dynamics model with the col- lected data from the target domain. Following prior works [36, 10], we employ an ensemble of Gaussian dynamics models (T's (s'[s, a) in an attempt to capture the epistemic uncertainty due the insufficient target domain samples. Given the source domain state-action pair (Ssrer Asre), generate an ensemble of fictitious states and obtain the corresponding values for each state-action pair, which we call fictitious value ensemble (FVE) Fir asre):
Furthermore,
state &eor induced by
M
Diar (Sares Ugro) == Q00(6 dellf (a i=1
(5)
In practice, the choice of in Eq. (4) is also nontrivial due to task-specific scales of the values and the non-stationary value function during training. We replace the absolute value difference with the likelihood estimation to address the problem. Specifically, we construct a Gaussian distribution with the mean and variance of FVE denoted as (Mean (Serer asre)), Var QTar (S derc))). Estimating the value of the source domain state as Vir (s're) == Q0(5 sree dsre)la',, (asre)ll,,er)((( (oot:_--((--a" ~-1(-1=) we introduce Fictitious Value Proximity (FVP) representing the likelihood of the source domain state value in the distribution:
A(Sare: asre, s'are) = P(V tar (5'are) Mcan ( QT (Sore: asre)), Var QT (Sare; Qsre))).
(6)
Based on the likelihood estimation, we utilize the rejection sampling to select fixed percentage data (i.e., 25%) with the highest likelihood from batch of source domain transitions at each training iteration. Specifically, we train the value function by optimizing the following objective:
1 2
T (lsaana) [ TQ,) + E (( [w(s, s') (Qo - TQo)²
+ arg min
where
a, s') = (A(s, a, s') Açx)
(7)
Ag% is the top E-quantile likelihood estimation of the minibatch sampled from source domain data, T represents the Bellman operator, and D Dtar denote replay buffers of two domains.
Consider the case when the agent can perform online interactions with the source domain, the training data mostly comes from the source domain, while we aim train policy for the target domain. Hence, exploring the source domain is essential to collect transitions that might be high-value concern- ing the target domain. Thus, we introduce an exploration policy E that maximizes the approximate upper confidence bound of the Q-value, i.e., T t- arg max, Ep~D..-UD.r. QUB(S, a) ((]))) where QUB(s, a) := max (Qo, (s, ? under the implementation with SAC [24] backbone. Im- portantly, the exploration policy E is separate from the main policy æ learned via vanilla SAC. E and π are used for data collection in the source domain and target domain, respectively. The optimistic data collection technique has been proposed for advanced exploration [11] while we utilize the technique in online dynamics adaptation setting.

---

5.2 Adaptation with Offline Dataset of Source Domain
So far, we have discussed the setting where the agent can interact with the souree domain to collect data actively. Nonetheless, simultaneous online access to the source and target domain might sometimes be impractical. In order to address the limitation, we aim to extend our method to the setting we refer to as Offline Source with Online Target, in which the agent can access a source domain offline dataset and a relatively small number of online interactions with the target domain.
To adapt VGDF to such a setting, we propose a novel theoretical result of the performance guarantee:
Theorem 5.1. Under the setting with offline source domain dataset D whose empirical estimation of the data collection policy is ma(as) == L 1(5.0),
En L(()) .lee Marc and Mear denote the source and target domain, respectively. We have the performance guarantee of any policy π over the two MDPs:
TlMar (T) 2 TIMarc (TT)
maaax
E
PMrre Pare (DrV(TD|I=)
B PiRAna [s(
(8)
(1 7)2
(a): polley regularization
(b): value difference
where ((s,a) Exrra [M
[QM (d,a))}
The proof of Theorem 5.1 is given in Appendix B. This theorem highlights the importance of policy regularization and value difference for achieving desirable performance. is worth noting that the policy regularization term can shed light on the impact of behavior cloning, which has been proven effective for offline RL [22]. Additionally, the value difference term has a similar structure to that of Theorem 3. Thus, we propose a variant called VGDF + BC that combines behavior cloning loss with the original selective data sharing scheme. The pseudocode is shown in Algorithm 2, Appendix A.
Experiments
In this section, we present empirical investigations of our approach. We examine the effectiveness of our method in scenarios with various dynamics shifts, including kinematic change and morphology change. Furthermore, we provide ablation studies and qualitative analysis of our method. Details of environment settings and the implementation are shown in Appendix D and Appendix E, respectively. Additional results are in Appendix F.
6.1 Adaptation Performance Evaluation
To systematically investigate the adaptation performance of the methods, we construct two types of dynamics shift scenarios, including kinematic shift and morphology shifts, for four environments (HalfCheetuh, Ant, Walker, Hopper) from Gym Mujoco [71, 7J. We use the original environment as the source domain across all experiments. To simulate kinematic shifts, we limit the rotation angle range of specific joints to simulate the broken joint scenario. As for morphology shifts, we modify the size of specific limbs while the number of limbs keeps unchanged to ensure the state/action space consistent across domains. Full details of the environment settings are deferred to Appendix D.
We compare our algorithm with four baselines: (i) DARC [17] trains the domain classifiers to com- pensate the agent with an extra reward for seeking Synammcc--onsisttee
behaviors; (ii) GARAT [14] trains the policy with an adversarial imitation reward in the grounded source domain via action trans- formation [25]; (iii) /W Clip (Importance Weighting Clip) performs importance-weige weighted bellman updates for source domain samples. The importance weights (i.e., Ptar(s') Is, a)/ Prre [s, a)) are approximated by the domain classifiers proposed in DARC, and we clip the weight to [10 [10-4, 4,1] to stabilize training; (iv) Finetune uses the 10S target domain transitions to finetune the policy trained in the source domain with 1M samples. Furthermore, Zero-shot shows the performance of directly transferring the learned policy in the source domain to the target domain, and Oracle demonstrates the performance of the policy trained in the target domain from scratch with 1M transitions. We run all algorithms with the same five random seeds. The implementation details given in Appendix E.
As the results in Figure show, our method outperforms GARAT and /W Clip in all environments. DARC demonstrates competitive performance only in the first two environments, while it does not work in other environments. We believe that the assumption of DARC does not hold in the failure cases due to the significant dynamics mismatch. GARAT fails in almost all environments, which we
7

---

Hal /Cheetah broken back thigh
Ant- braken hips
Walker braken right feat
Hopper broken joints
5000
3500
3500
3000
3900
3000
6080
2500
2500
2000
Return
5000
2000
2000
1500
2000
1500
1500
1000
1000
500
500
0.0
0.2
0.4
5.6
0.5
0.0
0.2
0.4
2.6
0.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
HalfCheetah no thighs
Ant short feet
Walker no right thigh
Hopper big head
6000
5000
4000
2000
Return
3000
3000
2000
2000
2000
2000
1500
1000
1000
0.0 022
04
0.0
0.2
0.4
0.0
0.2
0.6
0.0
022
0.5
Stops in Target Domain (x10*)
Stops in Target Domain 1x11%)
Stops In Target Domain (x10%)
Steps in Target Domain (x10*)
Zero-shot
Finetune
Figure 4: Adaptation performance in the target domain with kinematic mismatch (Top) or morphology mismatch (Bottom). Solid curves are average returns over five runs with different random seeds, and shaded areas indicate one standard deviation. We use data ratio I' = 10, which indicates all algorithms perform 10t online interactions with the source domain except Oracle.
believe is caused by the impractical action transformation from inaccurate dynamics models. The performance of Zero-shot suggests that the policies trained the source domains barely work in the target domains due to dynamics mismatch. Finetume achieves promising results and outperforms our method in two of eight environments. We believe that the temporally-extende behaviors of the pre-trained policy benefit learning in the downstream tasks with the assistance of efficient exploration. Nonetheless, our method is the only one that outperforms or matches the asymptotic performance of Oracle in four out of eight environments.
6.2 Ablation Studies
To investigate the impact of design components in our method, we perform ablation analysis on the ratio of transitions I, data selection ratio 5%, and the optimistic exploration.
Data ratio r. We employ different ratios of transi- tions from the source domain versus those from the target domain (T 5, 10, 20) for variants of our al- gorithm. The results shown in Figure 5 demonstrate that the performance of our algorithm improves with more source domain transitions when the number of target domain transitions is the same. This finding indicates that VGDF can fully exploit the reusable source domain transitions enhance the training efficiency concerning the target domain.
HaitCheetah- broken
HattCheetah- thighs
5000
4000
3000
3000
2000
1000
1000
[-
--
05 D4
6.8 1.a
0.6 527
0.5 0.6
Steps Dameir (x15P)
in Targel Domein (x10%)
Figure Effect of transition ratio
No Khhtth
Moppor
sea
Figure 7: Effect of the optimistic exploration tech- nique (i.e., T
Figure 6: Effect of data selection ratios 5%.
Data selection ratio 5%. We employ different data ratios (10% 25%, 50%, 75%) for the variants of our algorithm. Furthermore, we propose baseline algorithm Mix that learns with all source domain samples without selection (w,s,,, s's) = in Eq. (7)). The results, shown in Figure 6, indicate
8

---

Table 1: Results in the offline source online target setting. We evaluate the algorithms via the performance of the learned policy in the target domain and report the mean and std of the results across five runs with different random seeds.
Offline only
Symmetric sampling
H2O
VGDF + BC
halfcheetah- - broken back thigh
1128 + 156
2439 + 390
5761 +148
4834 + 250
halfcheetahh no thighs
361 + 39
2211 + 77
3023 + 77
3910 +60
hopper- broken hips
155 + 19
2607 = 181
2435
325
2785 75
hopper short feet
399 = 5
2144 = 509
868 73
3060 =60
walker- - broken right thigh walker- right thigh
1453 + 412
709 + 128
3743 +500
3000 + 388
975 + 131
872 301
2600
355
3293 +:306
that our algorithm performs robustly under various ratios within a specific range (e.g., 5% S 50%). Surprisingly, Mix performs exceptionally well in environments with kinematic mismatches but fails in scenarios with morphology shifts. We attribute this to the less significant dynamics shift induced by kinematic changes compared to morphology changes.
Optimistic data collection. To validate the effect of the optimistic exploration F. we introduce variant of method without B.. The results are shown in Figure 7. Removing the optimistic exploration technique results in performance degradation in three out of four environments concerning the sample efficiency, validating the effectiveness of the exploration policy.
6.3 Performance under Offline Source with Online Target
In this subsection, we extend our method the setting with a source domain offline dataset and limited online interactions with the target domain, investigating the performance of our method without online access to the source domain. We use the D4RL, medium datasets [20] of three environments (i.e., HalfCheetah, Walker, Hopper) for evaluation. We compare the proposed VGDF + BC (Section 5.2) with the following baselines: Offline only that directly transfers the offline learned policy via CQL [35] to the target domain; Symmetric sampling [3] that samples 50% of the data from the target domain replay buffer and the remaining 50% from the source domain offline dataset for each training step; H20 [49] that penalizes the Q function learning on source domain transitions with the estimated dynamics gap via learned classifiers. All algorithms have limited interactions with the target domain to 10y steps. The experimental details are shown in Appendix E.2. The results shown in Table demonstrate that our method outperforms the other methods in four out of six environments, indicating that filtering the source domain data with the value consistency paradigm is effective the offline-online setting.
6.4 Quantifying Dynamics Mismatch via Fictitious Value Proximity
Although the empirical results suggest that our method can adapt the policy in the face of various dynamics shifts, the degree of the dynamics mismatch can only be evaluated via the adaptation performance rather than be quantified directly. Here, we propose quantifying the dynamics shifts via the proposed Fictitious Value Proximity (FVP) (Section 5.1).
We approximate the FVP in Eq. (5) by calculating the average likelihood of batch of samples from the source domain by E[A(s, a, s') e')] s B E(s.a.")) A(s, a, s'). We show the approxi- mated FVP in Ant environments with kinematic or morphology shifts in Figure 8. We observe a significant gap between the FVP values of the paired domains, which suggests the target domain with the morphology shifts is "closer" to the source domain than the target domain with the kinematic shifts with respect to the value difference. FVP measured by value differences quantifies the long-term effect on the expected return. Such measurement can be regarded as way to quantify the domain discrepancies.
2:4
Figure 8: Quantification analysis of the approximated FVP in Ant environments.

---

7 Conclusion
This work addresses the online dynamics adaptation problem by proposing VGDF that selectively shares the source domain transitions from value consistency paradigm. Starting from the motivation example, we reveal the limitation of the prior dynamics- baseed method. Then we introduce novel value discrepancy perspective with theoretical analysis, motivated by the insight that paired transitions with consistent value targets can be regarded as equivalent for training. Practically, we propose VGDF and the variant for the offline source domain setting. Empirical studies demonstrate the effectiveness of our method under significant dynamics gaps, including kinematics shifts and morphology shifts.
Limitation and future directions. One limitation of our method is the reliance on the ensemble dynamics models. However, the recent work estimating the epistemic uncertainty with a single model [19] could be applicable. Furthermore, value-aware model learning [18] may improve our method by training dynamics models with accurate value predictions of the generated samples. Exploring the effectiveness of value consistency for generalizing across reward functions can be another direction for future research. Finally, validating the effectiveness of the data sharing method in the Sim2Real problem would contribute to the robotics community. The online interaction with the reality system could be risky, recent works [6, 70] can be integrated for safe online interactions.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (Grant No.62306242) the National Key R&D Program of China (Grant No. 20.2002//01000000 Shanghai Artificial Intelligence Laboratory, Shanghai Municipal Science and Technology Major Project Scientific Research Development Center in Higher Education Institutions by the Ministry of Ed- ucation, China (No.202 1001), Shanghai Engineering Research Center of Al and Robotics, Engineering Research Center of AI and Robotics, Ministry of Education, China. We would like to thank the anonymous reviewers for their valuable suggestions.
References
[1] lige Akkaya, Marcin Andrychowicz, Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.07113. 2019.
[2] Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang. Principled exploration via optimistic bootstrapping and backward induction. In International Conference Machine Learning, pages 577-587. PMLR, 2021.
[3] Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data, arXiv preprint arXiv:2302.0200: 029488, 2023.
[4] Mohammadamin Barekatain, Ryo Yonetani, and Masashi Hamaya. Multipolar: multi- mouti-source policy aggregation for transfer reinforcement learning between diverse environmental dynamics. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3108-3116 2021.
[5] Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pages 501-510. PMLR, 2018.
[6] Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety crities for exploration. In International Conference on Learning Representationss 2020.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1eXXvv/666.0/ 540, 2016.
[8] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real
10

---

world experience. In 2019 International Conference on Robotics and Automation (ICRA), pages 9973-8979, IEEE, 2019.
[9] Ching-An Cheng, Andrey Kolobov, and Alekh Agarwal. Policy improvement via imitation of multiple oracles. Advances Neural Information Processing Systems, 33:5587-5598, 2020.
[10] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamies models. Advances neural information processing systems, 31, 2018.
[11] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
[12] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullett. org, 2016-2021.
[13] Imre Csiszár and János Körner. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press, 2011.
[14] Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems, $3:3917-3929, 2020.
[15] Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto-tuned sim-to-real transfer. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1290-1296. IEEE, 2021.
[16] Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI.2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 161 1.02779, 2016.
[17] Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhut- dinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In International Conference on Learning Representations, 2020.
[18] Amir-massoud Farahmand, Andre Barreto, and Daniel Nikovski. Value-aware loss function for model-based reinforcement learning. In Artificial Intelligence and Statistics, pages 1486-1494.
PMLR, 2017.
[19] Angelos Filos, Eszter Vértes, Zita Marinho, Gregory Farquhar, Diana Borsa, Abram Friesen, Feryal Behbahani, Tom Schaul, André Barreto, and Simon Osindero. Model-value inconsistency as a signal for epistemic uncertainty. arXiv preprint arXiv:2112.04153, 2021.
[20] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
[21] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[22] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 11:20122-20144 2021.
[23] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018
[24] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
[25] Josiah P Hanna and Peter Stone. Grounded action transformation for robot learning in simulation. In Thirty-first AAAI conference artificial intelligence, 2017.
11

---

[26] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Aleny,, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. self-super policy adaptation during deployment. In International Conference on Learning Representations, 2020.
[27] Donald Hejna, Lerrel Pinto, and Pieter Abbeel. Hierarchically decoupled imitation for morpho- logical transfer. In International Conference on Machine Learning, pages 2020.
PMLR,
[28] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[29] David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[30] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances Neurul Information Processing Systems, 32, 2019.
[31] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810-21828 2020.
[32] Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning, pages $296-5295. PMLR, 2020.
[33] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research,
2013.
[34] Wouter M Kouw and Marco Loog. A review of domain adaptation without target labels. IEEE Iransactions pattern analysis and machine intelligence, 43(3))766-785, 2019.
[35] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179- 1191, 2020.
[36] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
[37] Alessandro Lazaric. Transfer in reinforcement learning: a framework and a survey. In Rein- forcement Learning, pages 143-173. Springer, 2012.
[38] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020.
[39] Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. Advances in neural information processing systems, 33:7754-7767, 2020.
[40] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
[41] Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment- imitation learning. In International Conference on Learning Representutions, 2019.
[42] Jinxin Liu, Zhang Hongyin, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. In International Conference on Learning Representations, 2021.
[43] Xingyu Liu, Deepak Pathak, and Kris Kitani. Revolver: Continuous evolutionary models for robot-to-robot policy transfer. arXiv preprint arXiv:2202.05244 2022.
[44] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference Learning Representations, 2018.
12

---

[45] Bhairav Mehta, Manfred Diaz. Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning, pages 1162-1176. PMLR, 2020.
[46] Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008.
[47] Fabio Muratore, Christian Eilers, Michael Gienger, and Jan Peters. Data-efficient domain randomization with bayesian optimization. IEEE Robotics and Automation Letters, 6(2):911- 918, 2021.
[48] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- reinforcement learning. In International Conference on Learning Representations, 2018.
[49] Haoyi Niu, Shubham Sharma, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, and Xianyuan Zhan. When to trust your simulator: Dynamics-aware hybrid offline-and andee-nnd--nieeeee reinforcement learning. XrXiv preprint arXiv:2206. 13464, 2022.
[50] Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 34:23283- 23295, 2021.
[51] Michael O'Connell, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural-fly -fyy enables rapid learning for agile flight in strong winds. Science Robotics, :(66)eebm6659, 2022.
[52] Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In ICLR (Poster), 2016.
[53] Xue Bin Peng, Marcin Andrychowiez. Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018
[54] Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pages 147-163. PMLR. 2017.
[55] Roberta Raileanu, Max Goldstein, Arthur Szlam, and Rob Fergus. Fast adaptation to new environments via policy-dynamies value functions. In Proceedings of the 37th International Conference on Machine Learning, pages 7920-7931, 2020.
[56] Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learn- ing robust neural network policies using model ensembles. arXiv preprint arXiv:1610.0120 2016.
[57] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.
[58] Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain random- ization via probabilistic inference for robotics simulators. arXiv preprint 1rXiv:1906.000 2019.
[59] Julian Schritwieserr, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604-600 2020.
[60] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.024338 2015.
[61] Jian Shen, Han Zhao, Weinan Zhang, and Yong Yu. Model-based policy optimization with unsupervised model adaptation. Advances in Neural Information Processing Systems, 33:2823- 2834, 2020.
13

---

[62] David Silver, Thomas Hubert. Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint 171.v:/71..0/815, 2017.
[63] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning. 3(1):9-44, 44, 1988.
[64] Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216-224. Elsevier, 1990.
[65] Richard S Sutton. Dyna, integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4): 1991.
[66] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[67] Richard Sutton, Csaba Szepesvári, Alborz Geramifard, and Michael Bowling. Dyna-style planning with linear function approximation and prioritized sweeping. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence, pages 928-536, 2008.
[68] Yunzhe Tao, Sahika Genc, Jonathan Chung, Tao Sun, and Sunil Mallya. Repaint: Knowledge transfer in deep reinforcement learning. In International Conference on Machine Learning, pages 10141-10155 PMLR, 2021.
[69] Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey, Journal of Machine Learning Research, 10(7), 2009.
[70] Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems, 34:13859- 13869, 2021.
[71] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[72] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprint arXiv: 1007.06/70333 2018
[73] Christopher JCH Watkins and Peter Dayan. Q-learning- Machine learning, 8(3):279-292, 1992.
[74] Annie Xie, Shagun Sodhani, Chelsea Finn, Joelle Pineau, and Amy Zhang. Robust policy learning over multiple uncertainty sets. arXiv preprint arXiv:2202.07013, 2022.
[75] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems,
2021.
[76] Wenhao Yu, C Karen Liu, and Greg Turk. Policy transfer with strategy optimization. In International Conference on Learning Representations, 2018.
[77] Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. arXiv preprint arXiv: 2017.
[78] Grace Zhang, Linghan Zhong, Youngwoon Lee, and Joseph J Lim. Policy transfer across visual and dynamics domain gaps via iterative grounding. arXiv preprint arXiv:2107.00339, 2021.
[79] Qiang Zhang, Tete Xiao, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Learning cross- domain correspondence for control with dynamics yycle-consistenyy In International Confer- ence on Learning Representations, 2020.
[80] Wenxuan Zhou, Lerrel Pinto, and Abhinay Gupta. Environment probing interaction policies. In International Conference on Learning Representations, 2018.
[81] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: very good method for bayes-adaptive deep via meta- learning. In International Conference on Learning Representations, 2019.
14

---

using in Miar A Algorithm Description
The pseudocode of VGDF is presented in Algorithm 1. We utilize SAC [24] backbone algorithm. We employ fixed entropy temperature coefficient in all experiments, demonstrating sufficient empirical performance. The training of the dynamics model ensemble follows prior works [10, 30] with the MIE loss. The calculation of the Fictitious Value Proximity follows Eq. (6) proposed in Section 5.1. Furthermore, the pscudocode of VGDF BC is presented in Algorithm We introduce the value-normalized tradeoff' between the behavior cloning loss and the policy gradient following the prior work [22].
Algorithm 1 Value-Guided Data Filtering (VGDF)
Input: Source domain Mares target domain Mrar, and transition ratio I' (= 10) (source vs. target). Initialization: Policy т. exploration policy T value functions {Qo, J==122 replay buffers (Dae ) dynamics model ensemble (To data selection ratio ç, batch size B, entropy temperature coefficient A.
1: for t= 1, do
2: # Interact with the source domain
Sample transition (Seror asres Tarcs s'rre) using _ in Marc
4: Dere f- Dare (Sercs aares l'are s sre)
5: # Interact with the target domain
6: % r == 0 then
7: Sample transition (Star; atars Ttars spar)
8: Dtar Dear U (Star, atary Ptary Stor)
9:
end if
10:
Optimize dynamics ensemble (To.) Jizl with Diar via Eq. (13)
Sample bave := {(s, a, r, r,) from Dare
12:
Sample bear {(s,((s,))) r, r,)))) s') Harr from Dtar
13:
Obtain Fictitious Value Proximity (FVP) {A(s, a, 8') via Eq. (6) for transitions in barc
14:
Obtain FVP quantile Aęx (A(s,a,s))
# Optimize value function with data filtering
0 0.2 + arg min Ebur [Qa. - TQQ,))
16:
0,
2B
1
17:
Ebyre (A(s,,,, > Aex) (Qo. - -QQ,.)
12B
5%
18:
# Optimize policies
1
F
arg max
Ebnar bare
max (Qo, (s, a). Qo, (s, (a) A7((TT
x²
2B
20:
arg 2B
Σ
branU bare
[min (s, Qo, (s, lawat(1)) XH[[]]]
21: end for
B Proofs of the Performance Guarantees
This section presents the proof of our main results. Specifically, we propose that the value discrepancy can be leveraged for the performance guarantee across different domains Lemma C.3, In Theorem B. we convert the performance bound induced by the value discrepancy into a novel form for offline souree domain setting.
Theorem B.1. (Performance bound controlled by dynamics discrepancy.) Denote the source domain and target domain with different dynamics as Marc and Mtar, respectively. We have the
15

---

Algorithm 2 Value-Guided Data Filtering + Behavior Cloning (VGDF+ BC)
Input:
Source domain offline dataset D Sre. target domain Miars max interaction steps with the target domain maxx and transition ratio (:= Derel 10) (source vs. target).
Taax
Initialization: Policy æ, value functions {Qa, H=1122 target domain replay buffer Dtars dynamics
model ensemble (To, Jiml+ data selection ratio batch size B, entropy temperature coefficient X, train repeat K, behavior cloning constant a.
for t= 1, .... maxx do
2: # IInteract with the target domain
3: Sample transition (Star; Otary Ttary star) using ग in Maar
4: Diar + Dear (Star, atars Ttar; star)
5: # Repeat training for K times per step
6: for k= 1, K. do
7: Optimize dynamics ensemble (Toz ) with Dear via Eq. (13)
8:
Sample bare == [(8, r, r, s') from Dare Sample bea := ((8, a, r; 8') ]or from Dtar
9:
Obtain Fictitious Value Proximity (FVP) (ss,,,,8)) via Eq. (6) for transitions in bave
11:
Obtain FVP quantile Ar% of {A(s, a, 8'}
12:
# Optimize value function with data filtering
0 arg min Emur (Qo. -TQu.)
13:
6:
2B
14:
Enere (A(s,a,8) > Agx) (Qe. TQo,)
J2B 5%
# Optimize policy with behavior cloning regularization
B a/ 2B Escarubere Immn (Qo, (s, a), Qo, a)) (m) ]}
16:
B
17:
arg max
Emeano Bare
min (Qa, (s, a). Qa (8, a) (l))))))))) XH[[]]
2B
1
(m())
B
19:
end for
20:
end for
performance difference of any policy evaluated under Mero and Miar be bounded as below;
27Tmmax
TlM (T) 2 T1M (T)
EEp... [Drv (Paro(-18, Ptar a))]
(1 7)2
Proof. We have
Msrc (TT) Mtar (Ir) = 1-9
E (e.3
Porc(s'ss,
Pear (8'1,, a) (5')ds'
(Lemma C.1)
[ (Pmrcls'ss,a) (sar(s' o/) (s')ds'
Epp (210)
J. [(Paro(o(ss,a) - (s') ds'
< E (prcls
?
5 1- 1 Bpan(an)) [/, [Psrc(s'ls,a)
Tmax E
2yTmax
(ptndc.)) (Drv (Parollss, Ptar|Pa (1 7)2
(9)
16

---

Theorem B.2. (Performance bound controlled by value difference.) Denote the source domain and target domain as Mare and Mear. respectively. We have the performance guarantee of any policy n. over the two MDPs:
7
Errac [Viia (s')] PP [VM.... (01)
IlMuu (T) 2 MI.arr (T)
Rapsare
1-7
Proof. We have
? E pprrloia) I (in. (8)) / Pear (s'(ss, as,, (o (d))ds)) (
(ore(T) Mear
(Lemma C.1)
? Eppaao
Epare [VMrr (5')] Bpear [VM (6')
S
? PPpene
Epano (Maar (s')] RRrrr V (s')]
1-9
Theorem B.3. Under the setting with offline source domain dataset D whose empirical estimation 1(s,a)
of the data collection policy is (ala)s) == Eo I(e) let Mare and Maar denote the source and target domain, respectively. We have the performance guarantee of any policy over the MDPs:
4r max
1
[((),a)]]] (10)
TMerr (TT) 2 TMMrr (TT)
(1 7)2 E8 pi Paro (DTV(TDll) - PM
18
1- where a() Ex [M - EPrant [QMan ( (8,
Proof. We have
TMaa (x) MM (T))
[MMar (TD) (TD) TMMer (T)
(IMano (TD) - TMMaa (T)
(0)
(b)
We have
(TD) -IMamm (T))
E
R Be''no(( [Quu. Eau of [Qi (s', [
sa-piam o / (NPorollssaa)
I-
E
Σ (To(a')s))<<(() QMmn (6,
?
E s,a-p 8 d'wPoro(-laia)
Σ (TD(d'ls') - -(a(ls)
Tmax
1
1
2
T maxx
E
Σ (Tl('')))) <a'ls'')
11
7)2
2r 2mxax
[Drv (TT(-11') T(-6/1)),
xo xaa-p 5 Parolal(l
7)2
and
(IM.no (TD) [Muur( (T))
1
Baaappp
E
(Lemma C.2)
1-y
2r max
2
(1 7)2
2,00p TD Mame s a' ('>PPre(((aaa)
[DTv(TD((ss') [(((s))
E T Mare
Ev Ba 1M... (s', a')] - Ed P'aratt [M (5', (s',a')]]) (Lemma C.3)
Combining the two inequalities above completes the proof.
17

---

C Proofs of Lemmas
This section provides proof of several lemmas used for our theoretical results. The first lemma is adopted from [44], and the proof is essentially the same the original paper. Lemma C.2 and Lemma C.3 support the derivation of the performance difference bound Theorem B.3.
Lemma C.1. (Telescoping Lemma, Lemma 4.3 in [44].) Let M1 == A, Pr.r, 7) and M2 (S, AS,A,PA 7)) be two MDPs with different dynamics Pi and Pz. Given a policy æ, let
MM (5, a) E. [VT, ENNP, [V (s')),
have
(m) - nMs -
7 (1-?)
Basaapie (s,
Proof. Define W3 as the expected return when executing क on Mi for the first steps, then switching to and M2 for the remainder. That is
W
7 E Ketee [r(s), a:)] = Besjar a 0= P
rr((s,, ax)
12
Then we have
Wo = Ex [r(st, = [T)
and Wx = Bx (r(st, =11, (T).
Thus we can obtain
20 (W WW)).
nM. TM. (T) (T)
(11)
5=0
Convert Wj and W as following:
W, R2 Pey [B [ VMn (s,+1)]
W = R2 + Ee [Be (1,++i)) [5]+
Plug back to 11 1 and we obtain
7M, (TT) 7Mo =
Σ
W s+1 - W
1=0
g E Eg
Evar, [s'] - -Brwn [V (s')
Miri
Eea-n Evaa [V (s')] - -Eswp (s')
(1 7)
(1 77)) Ee 9 (M,
(8, ())
Lemma C.2. (Extension of Telescoping Lemma.) Let M1 == (S,A, and Ma := (S, be two MDPs with different dynamics P1 and Pz. Given two policies Tı, T2, let
Ghim, (s, a) = Esrpar [QR, (s',aa)] Eg [Q
we have
nM, nM, (TT) - -m22(TT) (1 (1-7) s gg
(Mining
18

---

Proof. Define Wj as the expected return when executing Ti on M₁ for the first steps, then switching 42 and M2 for the remainder. That is
W Σ Y ar [r az)l Bechar P1 12
(r(( an)
Then we have
Wa -.,.a-paan [r(st, [:)] 1M, (x2);
and Wx Bspa-pmin [r(st, 11,M (T) (T1).
Thus we can obtain
(m1) = Σ
(W Wz).
(12)
j=U
Convert Wj and J+1 as following:
W = R2 + Bag Py Pi
Bag 1911 11 P2.02
[ Q (3++1]
W j++1+ R2 + E,y,a, Pi
[aj+1 Pi.m
[> + Qu. (+++):
Plug back to 12 and we obtain
MM, (TI) (T2)
(Wi+1 W
Σe
+
B. My
Eg a' [Q (s')
-Egra [Q (s a')]
7
E B
Eg Exratoi [Qi (5', a')]
Egraansra [Q (s',
(1 1 7)
E B
51
MMiii (0,
(1 7)
Lemma C.3. (Bound of 9 52
MI.M, (8, a).) Let
Mi,Ma (8, = Es F1 [QMi (8', .d)] - Ea (6',('',a))
have
gT (s, 2T Edaa (Drv(T) (-165) || ne(-ss))]
+ Bera [QM (s', a')] Er E Paima [00, (s', a')'")
Proof. We have
Muim, (s) a 10 (s', a')l
Exra 1 a')
= BB
(27 (s',a))
-Brra' [Q (8', aa))
(a)
+ EsjanAiin
[2 (s',a')) a')] - Ev o Pai [2 (5',a'))
(8)
19

---

For (a), we have
(a) = -
Σ (aalss - (all'ss'))ii
|oi (a'ls')) - =2(a'18') Tmax
< Ps
7
Tmax Ey'ns
Σ (ala'ss =2(a'1s\1
1 1-9
2 2fmax Bo (DVV (TI(+(s') x2(-s')))
1
For (b), we have
(b) Be Pl 1050 (s', ,d) Bs' (
< a' [ (5', a')] - -a P2.22 P2. Q (8', (s',
Adding these two bounds together yields the desired result.
D Detailed Environment Setting
D.1 Grid World
In the grid world environment, the agent obtains the X-Y coordination as the state and executes one of the four actions (Up, Down, Left, Right) at each time step. non-zero reward 1.0 is provided only if the agent reaches the goal. Each episode terminates when the agent reaches the goal or the episode length of 256 is reached. The source domain and the target domain of the grid world are shown in Figure 9. For each algorithm, the agent interacts with the source and target domains for 5e and 5e steps, respectively.
Figure 9: The source domain (Left) and the target domain (Right) of the grid world environments.
D.2 Mujoco Environments
To investigate the performance of the algorithm thoroughly, design eight environments based on four Mujoco [71] benchmarks from Gym [7] including HalfCheetah-v2, Ant-v4, Walker2I-v22, and Hopper-v2. For each benchmark, we propose two variants with kinematic shift morphology shift. We run all experiments with the original environment as the source domain and the variation environment the target domain. Detailed modifications of the environments are shown below, and the illustration of the environments is shown in Figure 10. For algorithms that access interactions with both domains, the agent interacts with the source and target domains for 106 and 103 steps. respectively.
Detailed modifications of the environments with kinematic shifts shown below:
HalfCheetah broken back thigh: We modify the rotation range of the joint on the thigh of the back leg from [-0.52, 1.05] to [-0.0052, 0.0105).
20

---

Source Domains
Target Domains with Kinematic shifts
Target Domains with Morphology shifts
Figure 10: Illustration of all environments, including all source domains (Top), all target domains with kinematic shifts (Middle), and all target domains with morphology shifts (Bottom).
Ant - broken hips: We modify the rotation range of the joints on the hip of leg 1 and leg 2 from [-30, 30] [-0.3, 0.3).
Walker - broken right foot: We modify the rotation range of the joint on the foot of the right leg from [-45, 45] -00.55, 0.45].
Hopper broken joints: We modify the rotation range of the joint on the head from [-150, 0] to [-0.15, 0] and the joint on foot from [-45, 45] [-18, 18).
Detailed modifications of the environments with morphology shifts are shown below:
HalfCheetah no thighs: We modify the size of both thighs. Detailed modifications of the xml file
<geom fromto-"O "0 -0.0001 -0.0001" name- "bthigh" size- 0. 046" type
="capsule" /> <body name=" bshin" pos="-0.00 0001 0 -0.0001">
<geom fromto="O 0 0000 o. 0001" name- fthighh size- "0.e="0.00 046" type-"
capsule"/> <body name=" "fshin" pos= o. 0001 0 0.0001
Ant short feet: We modify the size of feet leg and leg 2. Detailed modifications of the xml file
are:
Kgeom Cromto="O.0 0.0 0.0 0.1 0.1 0.0" name - same==left.ee "wee==eeett_eeeeeeeeee size="
08" type-" capsule"/>
geom fromto="0.0 0.0 0.0
-0.1
0.1 0.0" name "ame="right wwwperrrtttttaeeeeeeeeeeea size="
08 sypeulcappuue"..
Walker right thigh: We modify the size of thigh the right leg. Detailed modifications of the xml file are:
<body name="thigh" pos="0 0 1.05">
<joint axis="0 -1 0" name= name= ttiigh.joihhhttt pos="O 1.05" range= -55
0" type="hinge"/>
<geom friction="0.9" fromto= 0 1.05 o 1.045" name=" same="thigh.gemm
aize="0.05" cype="capaule"/>
<body name="leg" pos="O 0.35">
jjiint axis="O -1 0" name= leg-joint" pos="O 1. 0.04 range="
-15 type="hinge"/>
<geom friction="O..
fromto- 0 1.045 0 0.3" name- leg_geon"
size="0.00" sype=""cpssi capsule"
<body name="foot" 202 0">
21

---

<joint axis="0 -1 0" name="foot__jon pos="O 0 0.3" range="
455 45" hipe="hinge'/>>>>
geon friction="O. fro 0.3 0.2 0.3" name="
foot_geom" size= 00006" "ape-"capsule""" </body>
10
</body>
</body>
Hopper big head: We modify the size of the head. Detailed modifications of the xml file are:
geom friction="On 0. 9 fromto="O 0 1.45 1.05* name torrsorrrooooo geoo
size
=00. 125" typp="capsule""
E Algorithms and Implementation Details
E.1
Implementation Details
The details of our algorithm and baseline methods are specified as follows:
We first specify the implementation of the shared backbone algorithm SAC utilized in all algorithms. The policy and the value function are two-layer MLP with 256 hidden units using ReLU activation. The learning rate is 3e Discount y is set as 0.99 in all environments. The temperature coefficient is fixed as 0.2. The batch size 128. The smoothing coefficient of the target networks is 0.005. The training delay of the policy is set as 2. The replay buffer size is le
SAC:
VGDF: We use a five-layer MLP with 200 units as the dynamics model using Swish activation following prior works [10, 30J. The ensemble size is 7. We set the data selection ratio 5% as 25% in the experiments shown in Section 6. For each probabilistic dynamics model Tæ, (St+1. relse, = N (M (St, at). Eg, (st, a:)), i = 1,..., M, we train the model by maximizing the objective:
(ó.) = ((o1a), [[o (Se, ar)
(8)41, b0z (se, at) [Mm (se, at) - (8t+1, re)] )] + log det (,, (st, at)]-
(13)
The exploration policy is two-layer MLP with 256 hidden units. We warm-start the algorithm by utilizing samples from both domains without selection for the first 1e5 steps the source domain.
DARC: We follow the default configurations of the public implementation (https: //gi thub. wereoooleerreeercggggooooo eeeooglereeeerchh. weeeearccctteeeeeeeeeee htttt//eeeeeee/eeeee///eeeeeeeet htmmmmmatte/aaaaae
The domain classifiers Aysus (St, de, 8t41), Ut (st, a.) are trained by maximizing the cross-entropy losses:
J(bSAs) (s ) [log Iwsas (tarl St, at, st+1)]
E (( )~D.. [log(1 Iesas (tarl (t, az sc+1))]
J(HSAA B(c,at)DDarr [log Tos (tarls;, a:)] B(s. [log(1 gosa (tar)se, ax))]
Following the original implementation, we use the standard Gaussian noise for the domain classifier training. During training. reward correction Ar(st, ax) is augmented to the original reward r(st, az) of each source domain transition, i.e. i(st, 0e) == r(3,, ae) + (r at). The reward correction is calculated by: Ar(s;, ar) = log
JUSAS (tar s, a, s') quea (srcls, s. Awsae (src|s, 8²) Uoea (tar s, a)
We warm-start the algorithm by training with samples from both domains for the first 10 steps following the original implementation.
GARAT:
We use the tions (Supplemental in
author implementation with default configura- https://proceeiings htwproceeddingg.nerii....p http:cccppppe/2///////ee
Abt.rrat.tttttt............. We add the XML files of our customized environments to htt////v///aaaeee rl_gat/envs/asseets folder. We limit the extra interactions with the grounded source environments as 10 for fair comparisons with other algorithms.
22

---

public Table 2: Hyperparameters. "L" denotes the hyperparameter is not used in the algorithm. the same choice as the algorithm the first column.
denotes
Hyperparameters
VGDF
DARC
GARAT
IW Clip
Finetune
Hidden layers (Policy)
2
Hidden units per layer (Policy)
256
Hidden layers (Value)
2
+
(-
Hidden units per layer (Value)
256
Hidden layers (Classifier)
2
2
Hidden units per layer (Classifier)
256
256
Hidden layers (Dynamics model)
5
Hidden units per layer (Dynamies model)
200
Ensemble size
7
Learning rate
3e
Batch size
128
Fixed temperature coefficient
0.2
Target smoothing coefficient
0.005
Policy training delay
Buffer size
1e
Data selection ratio 5%
25%
Warm-start steps
1e
leb
1e
Importance weight clipping range Interactions with grounded sre environment
[1e_
1e5
Importance Weighting Clip (IW Clip): We use the domain classifiers same as DARC to calculate the importance weight w(s, a, s'). The importance weighting is calculated by:
(s'lss a) J0SAs (tar is a. s') Gos (srcls,
w(s, a, s') Pare(s'|s,a)
Iwsas (src 8, a, s") quea (tar|s,a)
where qusas and Svsa are the domain classifiers proposed in [17]. We use the importance weighing to reweight the value training with source domain samples. Specifically,
0 arg 2 ((aaane)) [wss, a, (') TTQ))
To stabilize training, we clip the importance weight between [le |1,--,110 11, same as the prior work [49].
Finetune: We first train policy in the source domain with 10t steps. Then transfer the policy to the target domain and further train the policy for 10 steps.
The detailed hyperparameters of all algorithms are listed in Table. 2, and we use the same hyperpa- rameters across all environments.
E.2 Implementation Details of the Offline-Online Experiments
To evaluate the performance of our algorithm in the offline source online target setting, we use medium datasets from D4RL [20] for three environments (i.e., HalfCheetah, Hopper, Walker). We use the same source domain offline dataset for each environment's two different target domains. For the algorithms performing online learning using offline data (i.e., Symmetric sampling, H20, VGDF + BC), we perform the online interactions with the target domain for 10 steps and use 106 source domain transitions, the training repeated for 10 times per step in the target domain. The details of the methods are specified as follows:
Offline only: We directly transfer the policy learned through CQL [35] with the source domain offline dataset. For the CQL implementation, we follow the suggested configurations
CQL implementation (https: / thub. com/tinkoff ai/CORL). We perform training for 10 steps with the offline dataset and report the zero-shot performance of the learned policy in the target domain.
Symmetric sampling [3]:
We perform the value function training by combining CQL. optimization (with offline transitions) and SAC optimization (with online transitions). For each training step,
23

---

sample 50% of the data from the target domain replay buffer and the remaining 50% from the source domain offline dataset. The CQL and SAC loss is computed with the corresponding transitions.
H2O [49]: We follow the original implementation that learns the classifiers to estimate the dynamics discrepancy across domains and perform the clipped importance weighting on the CQL loss on the source domain data. Same as Symmetric sampling, we repeat the training for 10 times per step in the target domain.
VGDF BC: We adapt VGDF to the Offline-Online setting by simply integrating the behavior cloning loss following (10). The training is repeated for 10 times per step with the target domain the same as the baseline methods. For the trade-off between the policy gradient and behavior cloning, we use the value-normane normalized regularization following the TD3 + BC [22] work and set the constant as 5. Furthermore, we remove the exploration policy proposed in Section 5.1 since the online access to the source domain is longer available in the offline-online setting.
Additional Experiment Results
F1 Quantifying Dynamics Shifts via FVP
In this section, we investigate whether the estimation of the value differences can quantify the difference across domains. Specifically, in different target domains of the same source domain, we demonstrate the estimation of FVP in two target domains. As the results show in Figure 11, the FVP differs in environments with different dynamics shifts (Kinematic or morphology). We observe that the FVP values in two target domains gradually approach each other in three out of four environments (HalfCheetah, Walker, Hopper), while the values in Ant remain relatively stationary. Furthermore, the FVP values in target domains with kinematic shifts are lower than those with morphology shifts across all four environments, which could result from the mismatched state space due to the limited joint ranges of robots in the target domain. Given the differences across different environments, we believe the FVP estimation could be used to quantify the domain differences.
HalfCheetah
Ant
Walker
Hopper
Approximated EVP F[A]
0.4
Morphology
Kinematic
0.3
0.2
0.1
0.0
0.2
0.4
0.6
0.8
1
0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps in Target Domain (x10')
Steps in Target Domain (x10))
Steps Target Domain (x109)
Steps Target Demain (x10')
Figure 11: Quantification analysis of the approximated FVP in all environments with different dynamics shifts. The dots are averaged values, and the error bars indicate the standard error across five runs.
F.2 Sensitivity to Ensemble Size
HalfCheetah- broken back thigh
HalfCheetah - no thighs
Hopper bww.brokkennnnnn
Hopper- big head
3000
3000
4000
3000
2500
Return
Return
2000
Return
2000
Return
2000
1500
1500
2000
1000
1000
500
500
0.0
0.0
0.0
0.6
Staps in Targot Domain (X16F)
Stops in Target Domain 1x10P)
Stops in Target Domain (x10*)
Staps in Tanget Domain (X10F)
Figure 12: Performance of the variants with different ensemble size values M. The results validate that a smaller ensemble size is sufficient to achieve competitive asymptotic performance compared to the variant with a large ensemble size in most environments.
24

---

We have introduced the dynamics model ensemble to capture the epistemic uncertainty induced by the limited samples from the target domain. However, training the ensemble of the dynamics model takes extra computation resources. Unlike prior works in model-based RI. [30, 61] that utilize the generated samples for training, we measure the value difference with the help of the generated samples. Therefore, we aim to investigate whether a smaller ensemble size is sufficient to achieve competitive asymptotic performance. Here we set the ensemble size as different values (M in the original implementation) and run experiments in four environments. As the results show in Figure 12, variants with a small ensemble size (e.g, M = 3 or = 5) can achieve identical asymptotic performance compared to the variant with a large ensemble size (e.g., M 7) in three out of four environments.
F.3
What about Importance Weighting via FVP instead of Rejection Sampling?
HaltChootah- broken back thigh
brokenn hips
Walker broken right foot
Hopper broken joints
5000
PM
3000
3500
3000
3500
2500
3000
Recurn
4000
3000
Return
2500
2500
Return
2000
2000
1500
2000
1500
1500
1000
1000
1000
1000
500
500
Steps Target Domain (x')
Steps
Domain (x10%)
Steps Target Domain (x10%)
Steps Target Domain (x10*)
HallCheetah thi hhs
Ant short feet
Walker thigh
Hopper big head
4000
3000
3000
2500
3000
2500
Retum
2000
Return
Retum
2000
2000
1000
1500
1000
1000
1000
1500
500
500
0.2 0.4
1.0
0.0 0.2
0.8
0.0
0.6
0.2 0.4
1.0
Stops In Target Domain (X110)
Steps Target Domain (x10")
Stops in Target Domain (x10%)
Stops in Target Domain (X100))
Figure 13: Performance of the variants with rejection sampling importance weighting technique. The results demonstrate that the original algorithm using rejection sampling outperforms the variant using importance weighting via FVP in almost all environments.
In the case of data selection based on the estimated FVP (fictitious value proximity in Eq. (6), one may wonder about using importance weighting via the FVP rather than rejection sampling, which might be sample-ine cample-inefficient due to the discarded partial data. Here we implement a variant of our algorithm that performs importance weighting with the estimated fictitious value proximity. Specifically, we train the value functions following:
01=1,2 arg min
1
Σ
[100 -QQ,))
2B
0;
((s.a.)))))
as, Elearsy A(s,a,s)) (Qo. TQo.))
Σ
2B
((8,ar;;"))
We compare the variant with the original algorithm using rejection sampling in all eight environments and demonstrate the results in Figure 13. The original algorithm using rejection sampling outperforms the variant with importance weighting in almost all environments. The accuracy of the value proximity depends the generated state and the value function. Thus, the estimation of FVP could be biased due to the inaccurate dynamics models and value functions in the early training stage, in which case naively utilizing the source domain samples weighted by the FVP can harm the policy performance concerning the target domain. In contrast, rejection sampling that only utilizes a small portion of source domain samples alleviates the negative effect of the source domain samples.
F.4
What about Data Filtering via Value instead of FVP?
Prior works have examined sharing data across tasks with different reward functions rather than dynamics [75]. To investigate whether selectively sharing data with high Q value can address the
25

---

HallCheetah- broken back thigh
Ant broken hips
Welker broken right foot
Hopper broken joints
3000
5000
3000
3000
2500
Retur
4000
3000
25:00
2500
Return
2000
1500
2000
1500
1000
1500
1000
1000
1000
500
500
0
Domain (x10f)
Steps Domain (x10P)
Steps Target Domain (xl0%)
Domain (x10%)
Holthhettah thi ghs
short
Walker- thigh
Hoppar big head
4000
3500
3000
3000
3000
2500
Retum
2000
2500
Return
Retum
2000
2000
2000
1500
1000
1500
1000
1000
500
500
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8 1.0
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2 0.4 0.6 0.8 1.0
Steps Target Domain (x11P)
Steps Target Domain (x10%)
Steps in Target Domain (x10%)
Steps in Target Domain (x110))
Figure 14: Performance of the variants that employ data filtering based Value or FVP. The results demonstrate that the original algorithm outperforms the variant using data filtering via Value in four of eight environments.
Walker broken right foot
Walker no right thigh
Hopper- broken joints
Hopper big head
3000
3500
DGDr
vGaar
3000
4000
2500
3000
2500
3000
2000
Retum
2000
Relur
2000
1500
Retum
2000
1500
1500
1000
1000
1000
1000
500
500
0
0
0.0
1.0
0.2
1.0
0.0 0.2
0.2 0.4
Steps Target Domain (x10")
Steps in Target Domain (x10f)
Steps in Target Domain (x108)
Steps in Target Domain (x10')
Figure 15: Comparison with the variant performing data filtering based on estimated dynamics discrepancies. The results demonstrate that the original algorithm outperforms the variant using data filtering via Value four of eight environments, validating the effect of the value consistency.
online dynamics adaptation problem, propose a variant of our algorithm that shares partial data with a relatively high Q value from the source domain. Specifically, we train the value functions following:
01 + arg min
Σ
[C00. - TQQ.)
2B
((,,0,
Σ
I (Qu > Q6%) (Qo. =Qo.))
2BB 5%]]
((sar))))
where Q5% is the top E-quantile Q value of batch of source domain samples. We set 5% as 25%, the same as our implementation. We compare the variant with the original algorithm all eight environments and demonstrate the results in Figure 14. The results demonstrate that the original algorithm outperforms the variant using data filtering via value in four of eight environments. Due to the dynamics mismatch, state-action pair from the source domain will lead to inconsistent states concerning two domains. Therefore, directly utilizing the transitions with high Q value without considering the consistency of the next state would provide counterfactual value target for the state-action pair, which can result in an improper value estimation for learning.
F.5 Comparison with Dynamics-guided Data Filtering
To investigate the effect of value consistency, we perform the ablation study by comparing VGDF to a variant that shares partial data based on dynamies discrepancies, ie.,, Dynamics-guidd Data Filtering (DGDF). Specifically, we estimate the dynamics discrepancy via the learned classifiers
26

---

Table Results in PyBullet environments. We evaluate the algorithms via the performance of the learned policy in the target domain and report the mean and std of the results across five runs with different random seeds. (# source, # target) denotes the number of source domain data versus the numbder of target domain data. HC and HP denote HalfCheetah and Hopper, respectively.
DARC
Finetune
VGDF
DARC
Finetune
VGDF
(# source, # target)
200k; 20k
1M, 20k
2004; 20k
1M, 100k:
1M, 100k
1M, 100k
PyBullet - HC
304 + 211
653 + 51
770 +:203
679 + 131
678
+ 38
808 1:89
PyBullet HP
73 + 20
240 + 189
957 1:39
99 +: 20
869 + 32
1006 -12
following the prior works [17. 49]. Same as VGDF, we share the source domain transitions whose estimated dynamics difference is smaller than the quantile value. We set the selection ratios as 25%,
the same as our implementation. The results demonstrate that the original algorithm outperforms the variant in three out of four environments, validating the superior effect of the value consisteney compared to the dynamics discrepancy.
F.6 Extended Results in Pybullet Environments
To investigate the generality of VGDF, perform additional experiments in ByBllet-HalfChheee and HyBullet-Hopper from PyBullet environments [12] which utilize Bullet as the physical engine instead of Mujoco. We first provide the details of the dynamics gap in the enviromments. In both environments, we regard the original environments as the source domains. In Hopperr we devised the target domain by increasing the torso size from 0.05 to 0.15, simulate the morphology change. In FrBBllet-Halfhheeee
we constrain the joint range of the front thigh from [-1.5, 0.8] 1-1.5, 0.4]. and the joint range of the front shin from [-1.2, 1.1] to 1.2, 0.1]. to simulate the broken joint scenario that is widely used in related works.
The results are shown in the Table 3, and we report the performance of all algorithms concerning different numbers of target domain samples. All results are averaged across five runs with different seeds. The results demonstrate that VGDF consistently outperforms baselines given different number of target domain data, demonstrating the generality our method.
27


Reviews:
[{"review_id": "6Z7ls8PN2L", "reviewer": "Reviewer_aEHs", "summary": "This paper focuses on the online dynamics adaptation problem where the agent has access to a large number of samples or an offline dataset from a source domain, and must adapt with a smaller number of samples in a target domain.\n\nTo address this problem, this paper introduces a framework for using a value discrepancy perspective. Specifically, they consider transitions with consistent value targets equivalent, even if they have different dynamics. This is different from prior works that focus purely on dynamics discrepancy, and encourage the agent to avoid state-actions with different dynamics. Towards this end, they show some theoretical analysis demonstrating a bound on policy performance given value consistency.\n\nGiven this perspective, they present the Value-Guided Data Filtering (VGDF) algorithm which uses value consistency to do selective data sharing from the source domain to the target domain.\n\nThe practical implementation of this algorithm involves learning an ensemble of gaussian dynamics models for the target domain. These different models are used to generate different fictitious transitions that can be used to estimate a gaussian distribution of potential values in the target domain for a given state-action pair from the source domain. Thus, the consistency of state-actions from the source domain are evaluated by Fictitious Value Proximity (FVP), which computes the likelihood of the estimated value of the source next state given the estimated gaussian distribution of potential values in the target domain. \n\nThey train a Q function for the target domain by using the samples from the target domain and samples from the source domain if they are in the top $\\xi$-quantile likelihood estimation of the sampled minibatch. They train an evaluation policy and exploration policy with SAC, with the main distinction being the exploration policy is trained to optimized the max of the 2 Q functions instead of the minimum. To tackle the setting where the agent only has an offline dataset collected in the source domain, they introduce a BC term to their optimization similar to TD3 + BC.\n\nThey evaluate their results in several Mujoco environments where the target domain introduces a kinematic shift or morphological shift to the robot. They generally find that their method outperforms their baselines in both the online and offline setting. The ablations demonstrate the benefits of the filtering by FVP and training the separate exploration policy. Finally, they demonstrate how FVP can be used to analyze different types of dynamics shifts in the target domain.", "questions": "Do you think your method will be effective in doing sim-to-real transfer? It seems to be a motivating example, but their are no results that directly indicate that your method will be better at sim-to-real transfer, than prior approaches.", "limitations": "No obvious limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well written, clear, and flows well. I believe an expert could reproduce the algorithm given the details in the paper.\n\nI believe the discussion on the value discrepancy perspective is an interesting contribution to the field that would interest other researchers.  I think in particular the theoretical results in section 4.2 and the motivating examples and figures in section 4.1 do a great job demonstrating the validity of this perspective.\n\nAs far as I am aware, VGDF is a novel and interesting algorithm that is well explained and justified by their value discrepancy perspective. The authors include good experimental results and ablations that demonstrate the effectiveness of their algorithm and specific design choices.\n\nAdditionally, I think the setting of doing online dynamics adaption from an offline dataset is a currently underexplored topic that should get more interest from the research community. The VGDF + BC is a seemingly simple yet effective extension of VGDF that seems to do well in this setting.", "weaknesses": "In my opinion, there are no major weaknesses in this paper, but I do have 2 small criticisms I would like to be addressed.\n\nI would appreciate comparisons in Section 6.3 to more offline RL algorithms that were designed for offline pretaining and online finetuning, like IQL.\n\nI believe there are many real-world scenarios where online samples are prohibitively expensive compared to simulated samples or previously collected offline data. Thus, I would appreciate more ablations with higher numbers for $\\Gamma$, but less online samples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback, and we will respond to each one of your questions below. \n\n**(W1)  I would appreciate comparisons in Section 6.3 to more offline RL algorithms that were designed for offline pretaining and online finetuning, like IQL.**\n\nWe would like to highlight the difference between the algorithms for Offline-Pretraining-Online-Finetuning (OPOF) and our algorithm that is designed for Online Dynamics Adaptation. Algorithms for OPOF typically do not consider the dynamics gap problem, the environment for offline training and online finetune are identical. In contrast, Online Dynamics Adaptation mainly focus on the dynamics gap problem, where a dynamics gap exists between the source domain and the target domain. Nevertheless, our algorithm is applicable to the offline-online setting. We compare VGDF-BC with IQL in OPOF setting with dynamics gap, the results shown in Table 1 verify the deficient performance of IQL when facing with the dynamics gap.\n\n|  Algorithm   | HalfCheetah – broken back thigh  | Hopper – broken joints |\n|  ----  | ----  | ---- |\n| IQL ($10^6$ offline pretraining + $10^5$ online finetuning)  | $2114 \\pm 141$ |\t$896 \\pm 134$ |\n| VGDF-BC ($10^6$ offline data + $10^5$ online data)  | $4834 \\pm 250$\t| $2785 \\pm 75$ |\n\nTable 1: Performance comparison between IQL and VGDF-BC in OPOF setting with dynamics gap.\n\n\n**(W2) I believe there are many real-world scenarios where online samples are prohibitively expensive compared to simulated samples or previously collected offline data. Thus, I would appreciate more ablations with higher numbers for, but fewer online samples.**\n\nWe agree that the online samples from real scenarios would be scarce and expensive. The ablations on the data ratio $\\Gamma$ of offline-online setting would be important to evaluate our method under the data-shortage scenarios. We have performed ablations on the data ratio $\\Gamma$ in the offline-online setting, and the results are presented in Table 2. These experiments demonstrate that the asymptotic performance of VGDF-BC is not significantly influenced by the higher $\\Gamma$ values with fewer online samples. This finding highlights the robustness and effectiveness of our algorithm even with limited online samples.\n\n| Data ratio $\\Gamma$  | Hopper – big head |\n| ----      |    ----           |\n| $\\Gamma = 10$ (# source = $10^6$, # target = $1\\times 10^5$)|\t$3060 \\pm 60$ |\n| $\\Gamma = 15$ (# source = $10^6$, # target = $6.7\\times 10^4$)|\t$3074 \\pm 74$ |\n| $\\Gamma = 20$ (# source = $10^6$, # target = $5 \\times 10^4$)|\t$2995 \\pm 36$ |\n\nTable 2: Performance of VGDF-BC with various data ratios $\\Gamma$.\n\n**(Q1) Do you think your method will be effective in doing sim-to-real transfer?**\n\nWe think that our data-sharing method is qualified to handle the sim2real problem. Actually, the most recent work [1] has demonstrated that simply sharing the experiences from tasks with different dynamics can be beneficial to the sim2real transfer. Current sim2real frameworks can be summarized by the four categories shown in Fig.1 of the main paper. We think that different methods might fit in different problem settings. For instance, the prevailing domain randomization for sim2real is essential when the target/reality is inaccessible. In contrast, our method focuses on scenarios when limited target domain data are available, which can lead to more directional and effective policy learning than domain randomization. However, our method does require online interactions with the target domain, which can result in safety issues in sim2real problems. Considering the advanced performance of our method in the dynamics adaptation setting, we believe integrating safe exploration techniques [2] into our framework would be an interesting future direction to devise novel sim2real frameworks. \n\nOnce again, we appreciate your valuable feedback and suggestions, and we will incorporate these points into the revised manuscript to strengthen the clarity and contributions of our work. If you have any further questions or concerns, please feel free to let us know.\n\n[1] Smith L, Kew J C, Li T, et al. Learning and adapting agile locomotion skills by transferring experience. arXiv preprint arXiv:2304.09834, 2023.\n\n[2] Thananjeyan B, Balakrishna A, Nair S, et al. Recovery rl: Safe reinforcement learning with learned recovery zones[J]. IEEE Robotics and Automation Letters, 2021, 6(3): 4915-4922."}, {"review_id": "4yrnLNi3VY", "reviewer": "Reviewer_Xugz", "summary": "Disclaimer: I found it challenging to fully comprehend the paper. This may be due to either inadequate clarity in the writing or my own limitations in understanding the subject matter.\n\nThis work introduces a method called Value-Guided Data Filtering (VGDF), which aims to enable online dynamic adaptation. By utilizing a set of state-action pairs from a source domain, VGDF selects relevant transitions to train a policy that performs well in a target domain with differing dynamics, all while not requiring the need for extensive interactions with the target domain. To assess the effectiveness of the proposed method, experiments are conducted on four Gym Mujoco environments that feature diverse, dynamic shifts, such as kinematic changes and morphological variations.", "questions": "I strongly recommend that the authors undertake a significant revision of their paper, particularly focusing on Section 4 and Section 5. Additionally, it is important to ensure that the paper is self-contained and does not necessitate reading the appendix. Therefore, I suggest that the authors incorporate additional details regarding the experimental setup from the appendix into the main body of the paper.", "limitations": "The authors seem to have adequately addressed the limitations.", "rating": 4, "confidence": 1, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The main results depicted in Figure 4 exhibit considerable strength and indicate that VGDF outperforms the baseline methods.\n- The proposed method is supported by a theoretical analysis presented in Section 4.\n- The authors have conducted a good number of analyses, including ablation studies and quantification of dynamic mismatch.", "weaknesses": "The main weakness of this paper lies in its lack of clarity in communication and presentation. Overall, I found it challenging to follow the arguments and ideas presented. Furthermore, it remains unclear what the significant contributions of this work are and the novel aspects of the proposed filtering mechanism. For instance, in lines 52-58, the authors attempt to summarize their contributions. However, empirically demonstrating the superiority of their method (contribution number 4) is not considered a contribution itself, as every claim requires proper evaluation.\n\nAnother weakness of this work is the limited breadth of the experimental analysis. While the main results in Figure 4 undoubtedly showcase excellent performance for VGDF in Gym Mujoco environments, it is uncertain whether this generalizes to other environments beyond that scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your valuable feedback, and we appreciate the opportunity to clarify and improve our paper based on your suggestions. \n\n**(W1) The main weakness of this paper lies in its lack of clarity in communication and presentation. ...**\n\nWe apologize for any confusion regarding the contributions of our work. To address this, we have provided a detailed description of the contributions as follows:\n1. In Section 4, we present a motivation example that highlights the limitations of prior methods using dynamics-based measurements. We provide theoretical analysis to interpret the results and introduce our novel value-based perspective. Unlike dynamics-based measurements that explicitly evaluate domain differences, our proposed perspective quantifies the transferability of transitions concerning the learning process itself. This enables superior performance in scenarios with significant domain differences, as demonstrated in Section 4.1 and Section 6.\n2. In Section 5.1, we derive the practical algorithm VGDF based on the insights proposed in Section 4.2.\n3. To extend the applicability of our method beyond the online source with online target setting, we introduce a variant of VGDF called VGDF+BC in Section 5.2, which is suitable for offline source with online target scenarios.\n4. In Section 6, we conduct extensive experiments to investigate the performance of our method. To simulate diverse dynamics shift scenarios, we design kinematic shifts and morphology shifts. Additionally, we perform ablations to analyze the effectiveness of our method.\n\n**(W2) Another weakness of this work is the limited breadth of the experimental analysis. ...**\n\nThe works for dynamics adaptation problems typically use Mujoco as a standard testbed to investigate the performance, including a large quantity of peer-reviewed papers [1,2,3,4,5,6,7,8,9,10]. Since the empirical investigation of dynamics adaptation problems requires simulating the dynamics shift scenarios, Mujoco turns out to be a well-suited platform thanks to the simplicity of changing the physical properties of the simulation model. Though we only perform experiments in Mujoco, we devise two different types of dynamics shifts (Kinematics and Morphology) which are different from prior peer-reviewed papers. Prior works typically only consider one type of dynamics shift, either Kinematics [1,2,3,4,5,6,7] or Morphology [8,9,10]. Thus, the empirical results over both types of scenarios further demonstrate the superiority of our method. The details of the environments are presented in Appendix D.\n\n**(Q1) I strongly recommend that the authors undertake a significant revision of their paper...**\n\nWe understand your concern regarding the script revision. After carefully considering your feedback and the feedback from other reviewers, we believe that our script is self-contained, and the main contents are presented in the main paper in a well-organized form. Deferring the experimental settings to Appendix is a common practice due to the page limitations. We have ensured that Sections 4 and 5 provide sufficient motivation, insights, and algorithm designs for a clear understanding of our work. Furthermore, each section starts with a conclusive paragraph for a comprehensible presentation. However, we would be grateful if you provide any suggestions for the revision of Sections 4 and 5.\n\n[1] Kumar S, Kumar A, Levine S, et al. One solution is not all you need: Few-shot extrapolation via structured maxent rl[C]. Advances in Neural Information Processing Systems, 2020, 33: 8198-8210.\n\n[2] Eysenbach B, Chaudhari S, Asawa S, et al. Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers[C]. International Conference on Learning Representations. 2020.\n\n[3] Shen Q, Li Y, Jiang H, et al. Deep reinforcement learning with robust and smooth policy[C]. International Conference on Machine Learning. PMLR, 2020: 8707-8718.\n\n[4] Lee K, Seo Y, Lee S, et al. Context-aware dynamics model for generalization in model-based reinforcement learning[C]. International Conference on Machine Learning. PMLR, 2020: 5757-5766.\n\n[5] Lee S, Chung S Y. Improving generalization in meta-rl with imaginary tasks from latent dynamics mixture[C]. Advances in Neural Information Processing Systems, 2021, 34: 27222-27235.\n\n[6] Ball P J, Lu C, Parker-Holder J, et al. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment[C]. International Conference on Machine Learning. PMLR, 2021: 619-629.\n\n[7] Mu Y, Zhuang Y, Ni F, et al. DOMINO: Decomposed Mutual Information Optimization for Generalized Context in Meta-Reinforcement Learning[C]. Advances in Neural Information Processing Systems, 2022, 35: 27563-27575.\n\n[8] Liu X, Pathak D, Kitani K M. REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfer[C]. International Conference on Machine Learning. 2022.\n\n[9] Chiappa A S, Marin Vargas A, Mathis A. DMAP: a Distributed Morphological Attention Policy for learning to locomote with a changing body[C]. Advances in Neural Information Processing Systems, 2022, 35: 37214-37227.\n\n[10] Hong S, Yoon D, Kim K E. Structure-aware transformer policy for inhomogeneous multi-task reinforcement learning[C]. International Conference on Learning Representations. 2021."}, {"review_id": "i3MIQxdSIG", "reviewer": "Reviewer_zSdL", "summary": "This paper considers a setting of online dynamics adaptation, where the goal is to train a near-optimal policy in the target domain using transition data from the source domain and the target domain with different dynamics. The authors propose to select source domain data to train Q-functions if the value discrepancy between the source and the target domain is minimal. The authors propose Fictitious Value Proximity (FVP) that represents the likelihood of the source domain state value and select source domain transitions with the top 25% quantile of FVP. The authors evaluated the proposed method on environments with different dynamics, including kinematics and morphology change.", "questions": "1. There seems to be a discrepancy between the method described in the paper and the code implementation of it. In the paper, VGDF selects the source domain samples based on the likelihood of the next state value $V(s’)$. On the other hand, in the code, VGDF selects the source domain samples based on the likelihood of $r + \\gamma V(s’)$, which approximates the current state value $V(s)$. In the code, the ensemble of dynamics models predicts the reward $r$ and the next state $s’$ in the target domain (line 286 in vgdf.py). These predictions are then used to compute Fictitious Value Ensemble (FVE) of $r + \\gamma V(s’)$ (lines 244 and 303 in vgdf.py) and Fictitious Value Proximity (FVP) of $r + \\gamma V(s’)$ on source domain samples. (line 261 in vgdf.py). This means that VGDF chooses source domain samples based on $V(s)$ instead of $V(s’)$, which contradicts the description in the paper. It is unclear why the authors chose to use $r + \\gamma V(s’)$ instead of $V(s’)$ in the code implementation.\n\n2. According to Fig. 7, using $\\pi_E$ seems insignificant. How exactly the exploration policy behaves in the source domain? I wonder whether maximizing $Q_{UB} = \\max\\{Q1, Q2\\}$ actually leads to optimistic exploration.", "limitations": "The authors included the limitations in the Conclusion section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper tackles the problem of generalizing policies across dynamics mismatch, which is significant in reinforcement learning.\n\n2. The writing and the clarity are good. Also, this paper includes a theoretical performance bound controlled by value difference to support the claim.", "weaknesses": "1. The proposed idea seems to be simple and obvious. VGDF wants to select source domain data with similar state transition dynamics for additional training data, and it uses a value function to measure the similarity of state transition dynamics.\n\n2. Although VGDF demonstrates superior performance baselines in some environments, it has comparable or inferior results to a fine-tuning method on the target domain in other environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback, and we appreciate the opportunity to clarify and improve our paper based on your suggestions. \n\n**(W1) The proposed idea seems to be simple and obvious. VGDF wants to select source domain data with similar state transition dynamics for additional training data, and it uses a value function to measure the similarity of state transition dynamics.**\n\nThank you for raising the question regarding the difference between value difference and dynamics difference and our approach to data filtering. We would like to clarify that dynamics difference refers to the discrepancy between a source domain transition and the corresponding target domain transition concerning the transition probabilities. On the other hand, value difference measures the discrepancy between the target domain's next state and the source domain's next state concerning the values. Thus, a minor dynamics difference of a transition can lead to a minor value difference, while the minor value difference does not necessarily lead to a minor dynamics difference. Filtering transitions based on small value differences is not proposed for obtaining transitions with minor dynamics discrepancies. \n\nAs discussed in Section 4.1, methods solely relying on dynamics discrepancies tend to provide overly pessimistic assessments of transitions, even though some transitions with significant dynamics differences could be beneficial for target domain policy training. Motivated by this observation, we devised an alternative approach, which involves filtering source domain transitions based on value discrepancies. These value discrepancies ensure that the selected transitions are equivalent to the corresponding target domain transitions concerning the value function training (as discussed in Lines 160-165). Additionally, the dynamics discrepancy considers only single-step shifts, whereas the value discrepancy takes into account the long-term influence of the transition. Theoretical analysis reveals distinct performance bounds for the two discrepancies, and empirical results validate the soundness of the value discrepancy perspective.\n\n**(W2) Although VGDF demonstrates superior performance baselines in some environments, it has comparable ...**\n\nAs for the performance of Finetuning, we believe that finetuning can be regarded as the most fundamental method of behavior transfer. It is possible to collect high-valued transitions by performing the pretrained behaviors in the target domain, which further benifits the policy training during finetuning. In the previous work [4], finetuning has even been proven to be stronger than several meta-learning algorithms. It is therefore not surprising that finetuning has comparable performance to VGDF in a few cases.\n\n**(Q1) There seems to be a discrepancy between the method described in the paper...**\n\nWe appreciate your observation regarding the implementation comparison of TD targets ($r(s,a) + \\gamma Q(s’,a’)$) and next-step values $V(s’)$. As we focus on the setting without reward function shifts ($r: S\\times A \\rightarrow \\mathbb{R}$), the reward of the current step can indeed be considered equivalent since they derive from the same state-action pair $(s_{\\text{src}},a_{\\text{src}})$. Moreover, we can use the $Q$-function to approximate the $V$-value by taking expectation on the current policy. However, to provide empirical evidence for the theoretical approximation, we perform additional ablation experiments on using TD targets versus next-step values. The results presented in Table 3 demonstrate that the performance is not significantly influenced by the implementation difference mentioned here.\n\n| Algorithm | HalfCheetah – broken back thigh | HalfCheetah – no thighs | Hopper – big head |\n| ----      | ----                            | ----                    | ----                    |\n| VGDF – TD target | $4735 \\pm 340$ |     $3579 \\pm 198$          | $3154 \\pm 121$          |\n| VGDF – V | $5016 \\pm 259$ |     $3785 \\pm 213$        |     $3187 \\pm 65$        |\n\nTable 3: Performance of VGDF using different implementation mechanisms for data filtering.\n\n\n**(Q2) According to Fig. 7, using $\\pi_E$ seems insignificant. ...**\n\nUsing the maximum of Q ensemble has been proved to obtain an optimistic estimation or an approximated upper bound of the value in prior works [1,2,3]. The goal behind employing this mechanism is to encourage the agent to collect source domain transitions that might be potentially high-valued concerning the target domain. While the improvement induced by the exploration technique might not be significant, the mechanism still yields non-trivial sample-efficiency improvement in specific environments.\n\n$\\quad$\n\nWe hope these explanations address your concerns and improve the clarity of our paper. If you have any further suggestions or questions, please feel free to share them with us. We value your feedback and are committed to addressing all aspects to enhance the quality of our work.\n\n\n[1] Ciosek K, Vuong Q, Loftin R, et al. Better exploration with optimistic actor critic[C]. Advances in Neural Information Processing Systems, 2019, 32.\n\n[2] Lee K, Laskin M, Srinivas A, et al. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning[C]. International Conference on Machine Learning. PMLR, 2021: 6131-6141.\n\n[3] Moskovitz T, Parker-Holder J, Pacchiano A, et al. Tactical optimism and pessimism for deep reinforcement learning[C]. Advances in Neural Information Processing Systems, 2021, 34: 12849-12863.\n\n[4] Zhao M, Abbeel P, James S. On the effectiveness of fine-tuning versus meta-reinforcement learning[C]. Advances in Neural Information Processing Systems, 2022, 35: 26519-26531."}, {"review_id": "j8c3wDdol1", "reviewer": "Reviewer_mz61", "summary": "The paper is concerned with the online dynamics adaptation problem, where an agent is tasked to generalize from a source domain with cheap access to a target domain with a limited online interaction budget. The approach filters what data from the source domain is used in the target domain based on whether the state-action pair will result in a state of similar value in both source and target domain. This approach is theoretically justified and motivated in a toy example. The authors show empirical results in modified versions of the Deepmind control suite.", "questions": "- I did not really understand why the fact that DARC learns overly pessimistic value estimations should be (as argued in section 4.1 and elsewhere) a direct consequence of DARC being dynamics-based rather than value-based? Would it not be possible to create a \"more forgiving\" version of DARC that, for example, uses a data-filtering mechanism rather than the (arguably over-pessimistic) reward shaping mechanism it currently uses? If I did not misunderstand the motivating example, I feel like it does not really disentangle the two aspects of (1) reward shaping vs. data filtering and (2) dynamics-based vs. value-based, but then the difference in results is mostly assigned to (2). The argument that value difference is more long-horizon than dynamics difference makes sense to me, though.", "limitations": "The limitations section is quite brief. I appreciate that space is very constrained, but maybe it would improve the paper to expand this a little for the camera-ready version.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper benefits from a relatively tight relation between theoretical motivation and practical implementation. The \"value discrepancy perspective\" is original in this context, and the authors carefully compare and ablate their method. The paper is easy to read and to understand. As sim2real transfer is a very real and pressing issue in robotics and control, I think the paper makes a significant contribution.", "weaknesses": "The paper could potentially benefit from additional experiments in environments that are not in the \"short-term control\" regime that is typical for the Deepmind control suite, but more long-horizion like physical manipulation scenarios or a more complex version of the maze that is used for the motivating example. Since in such scenarios, value assignment is generally harder to do correctly, it might be interesting to see how the performance of value-based data sharing is affected in such a case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback, and we will respond to each one of your questions as follows. \n\n**(W1) The paper could potentially benefit from additional experiments in environments that are not in the \"short-term control\" regime ...**\n\nWe appreciate your suggestion regarding additional experiments in long-horizon tasks to further investigate the performance of our method. Indeed, learning the value function in such environments can be challenging, and our method relies on the value function for data filtering. We think that addressing the difficulty of learning value functions in long-horizon tasks might be beyond the scope of the current paper. However, we believe that integrating additional mechanisms could be beneficial, and the simplicity of VGDF allows for these potential extensions to tackle the issue. For instance, we could explore using Episodic Control [1] or enhanced credit assignment techniques [2,3,4] to handle long-term credit assignment problems and improve performance in long-horizon tasks. These could be interesting future directions to enhance the applicability of VGDF in more complex environments.\n\n**(Q1) I did not really understand why the fact that DARC learns overly pessimistic value estimations should be ...**\n\nThank you for pointing out the difference between DARC and VGDF in terms of using reward-shaping mechanisms. To disentangle these two aspects, we have introduced a variant of DARC that performs data filtering with estimated dynamics difference in Appendix F.5. The results presented in Fig.15 demonstrate the superiority of the value-guided perspective in three out of four environments, highlighting the effectiveness of our value-based approach in comparison to dynamics-based methods.\n\n**(Limitation) The limitations section is quite brief. I appreciate that space is very constrained, but maybe it would improve the paper to expand this a little for the camera-ready version.**\n\nWe are grateful for your feedback regarding the limitations of our work. In the next version of this paper, we will extend the limitation discussion to address potential challenges and areas for improvement, as suggested.\n\n[1] Hu H, Ye J, Zhu G, et al. Generalizable Episodic Memory for Deep Reinforcement Learning[C]. International Conference on Machine Learning. PMLR, 2021: 4380-4390.\n\n[2] Arjona-Medina J A, Gillhofer M, Widrich M, et al. Rudder: Return decomposition for delayed rewards[C]. Advances in Neural Information Processing Systems, 2019, 32.\n\n[3] Raposo D, Ritter S, Santoro A, et al. Synthetic returns for long-term credit assignment. arXiv preprint arXiv:2102.12425, 2021.\n\n[4] Gangwani T, Zhou Y, Peng J. Learning guidance rewards with trajectory-space smoothing[C]. Advances in Neural Information Processing Systems, 2020, 33: 822-832."}, {"review_id": "vyw8Xts0Qx", "reviewer": "Reviewer_uy5c", "summary": "The authors propose a method that does domain adaptation for RL. Unlike other approaches that do this via domain randomization, offline interaction from the target env, or adjusting the sim with system ID from a small real dataset, the aim is to learn from a large source of sim transitions and a small number of **online** interactions.\n\nThe closest comparison to the work is DARC, which was also evaluated in a similar setting. DARC is from a class of methods that learns a reward correction term $\\Delta r$ that captures the change in environment dynamics between source and target domain, and estimates $\\Delta r$ using classifiers. This work suggests that DARC's estimation method is too conservative. In particular, given two tuples $(s,a,s_1')$ and $(s,a,s_2')$, if $V^\\pi_{tar}(s_1') \\approx V^\\pi_{tar}(s_2')$, then for the purposes of value estimation $s_1'$ and $s_2'$ can be considered similar, even if the individual states themselves are quite different. This is an example of a case where DARC could produce a larger reward penalty.\n\nThis work aims to estimate $V^\\pi_{tar}(s)$ from a small number of interactions in the target domain + many interactions in the source domain. To do so, an ensemble of dynamics models $T_\\phi(s'|s,a)$ is learned from target data. Sampling target state $s_{tar}'$ from the $T_\\phi(s'|s,a)$ family gives an ensemble of Q-value estimates $Q_{tar}(s,a)$, and this lets us model $V^\\pi_{tar}(s)$ as a random variable with some amount of uncertainty. The function is then updated with all target data, and some source data, where the source data is used only if it is measured as sufficiently \"in-distribution\" to the target environment. (In practice the authors use the top 25% most \"in-distribution\" source data.)", "questions": "Are there any results suggesting the accuracy of the value function $V$ during training? Or at least suggesting that any inaccuracies in $V$ are correlated during training?\n\nI would also appreciate more details on how the Gaussian dynamics models $T_\\phi$ were learned - given that the paper's central argument is to measure state distance by $V_{tar}(s)$, the method section for how $V_{tar}$ is learned seems oddly rushed.\n\n\nMisc questions:\n\nline 580 of appendix: it should be $W_{j+1} = ... + V(s_{j+1})$ instead of $V(s_j)$ right?\n\nline 565 of appendix: I'm not sure why the proof is 4 lines. Lemma C.1 gives $\\eta_1(\\pi) - \\eta_2(\\pi) = \\frac{\\gamma}{1-\\gamma} E[ E_{P_1}[V_{tar}(s')] - E_{P_2}[V_{tar}(s')] ]$. The end of Theorem B.2 is equivalent to, $\\eta_1(\\pi) - \\eta_2(\\pi) \\le$ \"the outer expectation, if you take the absolute value of the insides before integrating\". So I'm not sure why there's a step where the $P_{src}$ and $P_{tar}$ terms are shuffled around before taking the absolute value, seems like needless notation unless I'm missing something.", "limitations": "Limitations seems addressed fine.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I found the theoretical arguments reasonably clear, modulo a few typos (see later section), and on double-checking them, everything looks correct.\n\nI think the core idea makes sense: that for the purposes of learning, if you assume your value estimate $V$ is accurate enough, then it is sufficient to compare environments based on your value estimate $V(s_{src})$ vs $V(s_{tar})$ since the value function encapsulates all future variation. This idea is compelling.", "weaknesses": "In terms of references, there are some approaches that use value function equivalence as an objective for learning their manipulable sim (i.e. RL-CycleGAN, if you view learning an image-to-image GAN as a manipulable sim).\n\nMy more specific reservation is with the comparison to DARC, the most competitive of the baselines. The GridWorld example given shows an environment where DARC is overly pessimistic in bottleneck states, and fails to explore outside of that region. DARC has theoretical guarantees of correctness, so what is going on here? Perhaps in practice, the theory is too conservative (the field has seen similar things with TRPO / PPO using less-conservative approximations of theory).\n\nIn this scenario, my natural response would be to not to derive a new method, but to try weighting the reward correction term $\\Delta r$ in DARC by some hyperparameter. It is not very theoretically principled but would probably handle the pessimism alright. Does this approach meaningfully improve on DARC if the conservativism term was weighted? It's unclear.\n\nSetting this aside for now, if $V$ is learned well, then this paper (VGDF) makes sense. But to me this seems like a very strong \"if\" to make. It's a little chicken and egg, where $V$ should only be learned well if we did not need the source data in the first place? To determine the closest section of source data, we must already learn a 1-step dynamics model of the target environment $T(s_{t+1}|s,a)$, and estimate $V_{tar}(s')$ to estimate the future value from $s'$. If both those pre-requisites are true, it feels like we ought to just use a model-based method instead.\n\nAs implemented, the paper always uses the 25% closest segment of the source data, but there is no guarantee this 25% closest segment is actually that close to the target data. The ablations in Figure 6 suggest that the separation function is doing something, but I am a bit concerned that performance does not seem to differ much for many different thresholds $\\xi$.\n\nI am also not so fond of the exploration policy $\\pi_E$. It makes sense but I believe the baselines do not use a separately trained exploration policy (please correct me on this if wrong). It doesn't seem like $\\pi_E$ is a major component based on the ablation run, so this is a lesser concern for me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback, and we will respond to each one of your questions as follows. (**W and Q denote weakness and question, respectively.**)\n\n(W1) Thank you for reminding us of the related work RL-CycleGAN, which utilizes value equivalence for sim2real visual control. We will add RL-CycleGAN as one of the references. However, it's important to note that our work differs from RL-CycleGAN in several aspects: (1) We specifically focus on dynamics shifts between domains, while RL-CycleGAN generalizes across different visual or state spaces. (2) RL-CycleGAN requires bidirectional consistency, whereas our method solely addresses single-directional adaptation.\n\n(W2) In fact, DARC's theoretical guarantee requires an assumption, there is at least one policy that is near-optimal in both source and target domain. (Assumption 1 in DARC paper). This assumption constrains the method to the scenarios, where the gap between the source domain and the target domain cannot be too significant, as discussed in Lines 152-159 of our paper. In the toy task of Section 4.1, the assumption does not hold. That is, there is no policy that can complete navigation in both the source and target domains. Thus, DARC fails due to overly pessimistic assessments of the bottleneck transitions.\n\n(W3) Regarding the idea of reweighting the correction reward to address the problem of DARC, we agree that it might alleviate the pessimistic assessments, but it may not entirely resolve the issue. As discussed in Lines 152-159, the reward correction of DARC (i.e., $\\Delta r(s,a,s’) = log (P_{tar}(s’|s,a)/P_{src}(s’|s,a))$) tends to be $-\\inf$ when the source domain transition is not likely in the target domain (i.e., $P_{tar}(s’|s,a)\\approx 0$). Thus, even if we use a weight $0< \\alpha < 1$ to scale down the reward correction, DARC still produces negative reward correction (penalties) for transitions, which indicates the pessimistic assessments are unchanged even if the transitions are transferable from the value-discrepancy perspective. An alternative solution is to replace the reward correction of DARC with dynamics-based data filtering, which removes the pessimistic effect on the reward, as discussed in Appendix F.5. However, the results in Fig.15 demonstrate that the method still underperforms VGDF.\n\n(W4) Theoretically, perfect data filtering does require an accurate value function, and the accurate value function might only be learned with pure target domain data. However, we believe that a small number of target domain data can provide a good initialization for data filtering. In addition, the source domain data is selected only when the value equivalence holds, meaning the selected source domain data is nearly equivalent to the corresponding target domain data, concerning the value function training, as discussed in Lines 160-165. \nRegarding the accuracy of the dynamics model, we acknowledge that it also influences data filtering. However, unlike model-based methods (e.g., MBPO) that use the generated samples for efficient training without considering their validity, VGDF only compares the values of them with others without training with them. This alleviates the model exploitation problem common in model-based works. Furthermore, many model-based methods perform multi-step generation starting from collected samples, requiring a high level of prediction capability. In contrast, VGDF only requires the one-step prediction of the dynamics model. Thus, VGDF can tolerate a less accurate dynamics model, while the model-based might fail given the same model.\n\n(W5) We agree using 25% of batched samples does not guarantee value consistency, and the result of Fig. 6 verifies that it is helpful to restrict the usage of source domain data in some environments. The vertical axis of Fig. 6 is scaled too small, which makes the actual performance difference less obvious. In fact, there will be hundreds of score gaps at different thresholds in the two Half-cheetah environments. Thus, the threshold still influences performance to a non-trivial extent.\n\n(W6) It is true that the baselines do not incorporate optimistic exploration. Using the maximum of Q ensemble has been proved to obtain an optimistic estimation or an approximated upper bound of the value in prior works [1,2,3]. Here we intended to encourage the agent to collect source domain transitions that could be potentially high-valued concerning the target domain. Although the asymptotic performance improvement induced by the exploration technique is insignificant, the mechanism still produces non-trivial sample-efficiency improvement in specific environments.\n\n(Q1) Here we analyze the value accuracy by plotting the scatters involving predictions and ground truth Monte Carlo returns. Fig 1 in the **Global Rebuttal PDF** demonstrates that the value functions at different training stages approximate the expectation of the ground truth returns, which further validates that filtering source domain data via our proposed value consistency does not significantly influence the accuracy of the value functions.\n\n(Q2) The dynamics models are learned through MLE (Eq. 13 in Appendix E.1) with all samples collected from the target domain, following prior model-based works [4,5].\n\n(Misc Questions) Thank you for pointing out the presentation flaws in the theoretical parts. We will ensure to rectify these issues in the next version of the paper. We really appreciate your meticulous review.\n\n[1] Ciosek K, et al. Better exploration with optimistic actor critic. NIPS 2019.\n\n[2] Lee K, et al. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. ICML 2021.\n\n[3] Moskovitz T, et al. Tactical optimism and pessimism for deep reinforcement learning. NIPS 2021.\n\n[4] Janner M, et al. When to trust your model: Model-based policy optimization. NIPS 2019.\n\n[5] Chua K, et al. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. NIPS 2018."}]

Author rebuttal:
We appreciate valuable feedback and suggestions from all reviewers. If you have any further suggestions or questions, please feel free to share them with us. We value your feedback and are committed to addressing all aspects to enhance the quality of our work.

Meta-review: This paper presents a method for adapting a learned policy to a new domain using samples from the old domain. Past works in this problem space have used a data filtering approach for policy adaptation based upon domain dynamics similarity. This work instead proposes to filter based on value. The idea is motivated with simple theoretical analysis using lower bounds on policy performance in the target domain, practically instantiated, and then robustly evaluated on policy adaptation tasks created from MuJoCo benchmark domains.

Main Stengths: Reviewers praised clarity, reproducibility, novelty, strength of the empirical evaluation, and effort put into including ablation studies as well as benchmarking performance relative to baselines. I also find the method to be novel and find the idea of filtering based on value to be intriguing. Motivating this from a lower bound on policy performance is a compelling argument.

Main Concerns: in some cases the proposed method performed on-par with baselines. While some reviewers found the paper clear, others did not. Other minor criticisms were raised that can be addressed for the camera-ready.

Overall the paper was mainly viewed favorably by the reviewers and I believe the small noted concerns can be addressed in a simple revision. I encourage the authors to carefully consider the reviewer comments for the camera-ready and I recommend acceptance.
Decision: Accept (poster)