Cross-Domain Offline Policy Adaptation with Optimal Transport and Dataset Constraint
Authors: Jiafei Lyu
Mengbei Yan
Zhongjian Qiao
Runze Liu
Xiaoteng Ma
Deheng Ye
Jing-Wen Yang
Zongqing Lu
Xiu Li
Venue: ICLR 2025 Poster (2025)
Keywords: cross-domain
reinforcement learning
offline RL
optimal transport
Primary area: reinforcement learning
TL;DR: We propose a simple yet effective method that leverages optimal transport and support constraint for efficient cross-domain offline RL.

Abstract: We explore cross-domain offline reinforcement learning (RL) where offline datasets from another domain can be accessed to facilitate policy learning. However, the underlying environments of the two datasets may have dynamics mismatches, incurring inferior performance when simply merging the data of two domains. Existing methods mitigate this issue by training domain classifiers, using contrastive learning methods, etc. Nevertheless, they still rely on a large amount of target domain data to function well. Instead, we address this problem by establishing a concrete performance bound of a policy given datasets from two domains. Motivated by the theoretical insights, we propose to align transitions in the two datasets using optimal transport and selectively share source domain samples, without training any neural networks. This enables reliable data filtering even given a few target domain data. Additionally, we introduce a dataset regularization term that ensures the learned policy remains within the scope of the target domain dataset, preventing it from being biased towards the source domain data. Consequently, we propose the Optimal Transport Data Filtering (dubbed OTDF) method and examine its effectiveness by conducting extensive experiments across various dynamics shift conditions (e.g., gravity shift), given limited target domain data. It turns out that OTDF exhibits superior performance on many tasks and dataset qualities, often surpassing prior strong baselines by a large margin.

Published as a conference paper at ICLR 2025
CROSS-DOMAIN OFFLINE POLICY ADAPTATION WITH OPTIMAL TRANSPORT AND DATASET CONSTRAINT
Jiafei Lyu's ; Menggei Yan', Zhongjian Qiao', Runze Liu', Xiaoteng Ma², Deheng Ye Jingwen Yang', Zongqing Lu,,5, Xiu Lilet
'Tsinghua Shenzhen International Graduate School, Tsinghua University "Department of Automation, Tsinghua University, Teeneet
"School of Computer Science, Peking University, 'Beijing Academy of Artificial Intelligence Lvjf200 @ma i s. t tainghua...... cr, li li.xiudsz. t tigghuu edu. cn
ABSTRACT
We explore cross-domain offline reinforcement learning (RL) where offline datasets from another domain can be accessed to facilitate policy learning. How- ever, the underlying environments of the two datasets may have dynamics mis- matches, incurring inferior performance when simply merging the data of two domains. Existing methods mitigate this issue by training domain classifiers, using contrastive learning methods, etc. Nevertheless, they still rely amount of target domain data to function well. Instead, we address this problem by establishing a concrete performance bound of a policy given datasets from two domains. Motivated by the theoretical insights, we propose to align transitions in the two datasets using optimal transport and selectively share source domain samples, without training any neural networks. This enables reliable data filtering even given few target domain data. Additionally, ization term that ensures the learned policy remains within the scope of the target domain dataset, preventing from being biased towards the source domain data. Consequently, we propose the Optimal Transport Data Filtering (dubbed OTDF) method and examine its effectiveness by conducting extensive experiments across various dynamics shift conditions (e.g., gravity shift), given limited target domain data. It turns out that OTDF exhibits superior performance on many tasks and dataset qualities, often surpassing prior strong baselines by large margin.
large
introduce dataset regular-
INTRODUCTION
Alice used to play tennis without any exposure to other ball sports. Recently, the tennis court needs maintenance, and Alice ends up playing badminton with Bob. Alice quickly gets familiar with this sport. As depicted in this example, human beings are capable of swiftly adapting their policies to structurally similar tasks. We expect the same phenomenon to be observed in reinforcement learning (RL) agents. To that end, we aim at achieving better performance the target domain with a limited budget by accessing source domain (e.g.. simulator) where the two domains may have distinct transition dynamics. Such setting referred as the policy adaptation problem (Xu al., 2023).
Many works focus on handling the online policy adaptation problem (Xu et al., 2023; Lyu et al., 2024a; Niu et al., 2022) where either the source domain or the target domain is online. Instead, we are interested in offline policy adaptation, or the cross-domain offline RL problem (Wen et al., 2024) (i.e., both domains are offline), since online interactions can be expensive and even dangerous. The cross-domain offline RL setting is common in practice. For example, one research team gathered historical trajectories of humanoid robot. However, over time, the robot's s physical body parts may degrade get damaged during operation. That said, the dataset collected by then would differ from the past datasets in transition dynamics. It becomes a typical cross-domain offline RI. scenario if one decides to utilize past data for training policies. Recent advances in cross-domain offline RL include learning domain classifiers to estimate the dynamics gap (Liu et al., 2022a), filtering source domain data based on mutual information (Wen et al., 2024), etc. Unfortunately, these methods still run on
Work done while working as an intern Tencent. Corresponding Author.

---

Published as a conference paper at ICLR 2025
Dalaset Regularization
Source domain dataset
Optimal Transport
Source Data Filtering
Offline
Target domain dataset
Sample
An overview of our proposed framework. We first align source domain data and target domain data via the Wasserstein distance. Then adopt the solved optimal coupling for selectively sharing source domain transitions with the downstream offline RL algorithms. We further introduce a regularization term encourage the learned policy lie in the support region of the target domain.
Figure 1:
comparatively large target domain dataset. This can be problematic since offline RL methods like ReBRAC (Tarasov et al., 2024) can achieve strong performance on it, downgrading the necessity of extra source domain dataset. Hence, we consider the offline policy adaptation problem given very limited target domain data, with which the single-domain offline RL methods often struggle.
In light of such challenge, first theoretically characterize the performance bound of policy in the true target domain and the empirical source domain, which is related to the deviations of the learned policy against the behavior policies in the source domain dataset and the target domain dataset, and the dynamics gap between the two domains. The theoretical results highlight the ne- cessity of selectively sharing source domain data to close the performance gap rather than simply merging data from two domains. Given that only a few target domain data are accessible, we resort to optimal transport to find optimal alignments between data from two domains without training any neural network. We then the similarity measure between transition in the source domain dataset and the entire target domain dataset to filter out dissimilar source domain data. This enables reliable source domain data selection regardless of the target domain dataset size. However, it is still insuf- ficient to fulfill efficient offline policy adaptation, since the performance bound also controlled by the deviation of the learned policy against data-collecting policies in both domains. To mitigate this, we further introduce a policy regularization term that constrains the learned policy from executing out-of-distribution (OOD) actions outside the support region of the target domain dataset. We name our method Optimal Transport Data Filtering (OTDF) and summarize its framework in Figure I.
We evaluate OTDF upon various D4RL (Fu al., 2020) datasets with different types of dynam- ics shifts (e.g. gravity shift), given limited target domain data. Empirically, we demonstrate that OTDF achieves superior performance across numerous tasks and with varied source or target do- main dataset qualities, often outperforming recent strong baseline methods by large margin. To ensure that our work is reproducíble, our code available at http:://ithuu..cmm/tks////// comddmkssf///mmm//ftt
2 PRELIMINARIES
Reinforcement Learning (RL). RL problems can be formulated by Markov Decision Process P,r.7) AS,,,, where S denotes the state space,
(MDP) M, which is defined by the 5-tuple M = (S,A, A is the action space, P represents the transition probability, r(s,a) S A A- is the scalar reward signal, and E [0, 1) is the discount factor (Sutton & Barto, 2018). We assume that the reward signals are bounded, i.e., |r(s, a)l | Tmaxx Vs,a. The objective of the RI. agent policy π to maximize the expected discounted cumulative return E- [ (f((st, 0z)]. We assume that we have access to a source domain Mare = (S, Parcs r, 7) and a target domain Miar = (S, A, Ptar, r, 7), where the two domains only differ in their transition dynamics. We denote the normalized probability that a policy N encounters the state action pair (s, a) in the domain M as (M((s,) a) == (1 7) Er 7 "PM, (a)s) where P (s) denotes the probability that the policy т encounters the state at timestep M. Then, the performance of a policy π in M can be formulated (T) Egaapp [r(s, a)). We denote Pare P Ptar P
learn
2

---

Published as a conference paper at ICLR 2025
In offline RL, the agent only get access to static dataset D = {(s;, d,, Fa. 85+1)) where N = | is the dataset size. The goal of cross cross-domain offline RL is to improve the performance of the agent in the target domain by leveraging the mixed dataset Dixx = Dare J Dtars where Dare is the source domain dataset and Dtar the target domain dataset. We further define the empirical MDP in the dataset D as M := (S, Prr,)). P denotes the empirical transition probability in the dataset and P(s'|s, a) 0 for all OOD transition pairs. That said, have empirical MDPs in cross-domain offline RL, Mare with dynamics P PMm.e MMar with dynamics PM...
Optimal Transport (OT). OT (Cuturi, 2013; Peyré Cuturi, 2019) is a popular method to mea- sure distribution discrepancy, The Wasserstein distance between two discrete measures Pax 1 Dr=1 òz, and My = Ei= oy, is defined as:
W(ua: My) = min PEM
Clre Be Me.05
(I)
===1=
where C' is cost function, M {u E Raxn', : -1, #1 is the set of the coupling matrices, &g denotes a Dirac delta measure for Pt,e' the t-th row, t'-th column element in μs, and n, n' are the sizes the measures Mzz My, respectively. Solving Equation I results in an optimal coupling μ² that depicts an alignment between the samples in Mz and Myy-
3 PROSS-DOMAIN POLICY ADAPTATION UNDER LIMITED SAMPLES
In this section, we start by theoretically unpacking the performance difference between a policy in the true target domain and the empirical source domain. Motivated by thcoretical insights, for- mulate a novel objective for cross-domain offline RL with optimal transport and support constraints. Moreover, introduce our practical algorithm fulfill dynamics- aware offline policy adaptation.
3.1 THEORETICAL ANALYSIS GIVEN TWO OFFLINE DATASETS
Since we involve source domain offline data to facilitate policy learning in the target domain, is vital investigate how source domain data can affect the performance of the learned policy in the target domain. To that end, we derive the performance bound of any policy π in the target domain and the empirical source domain below, where DTv (pl (g) denotes the total variation deviation between the probability distributions p. q. Due to space limits, all proofs are deferred to Appendix B.
Theorem 3.1. Denote the empirical policy distribution in the offline dataset Dave from source do-
Lo 1(s.0)
main Mare and the offline dataset Diar from target domain Miar DD
and
EDme I(s)
Dear = Eo 1(s,a)
Emme I(s) respectively. Denote C1 = - then the return difference of any policy
between the empirical source domain Mare and the true target domain Mtar is bounded:
J. JMn (T) J M... (T)) -C1 E Dare Mure
(Drv(TDe.lIT) [[))
C1 Bppear PM [Dry (ry((Tr DT. [z)
(b): target policy deviation
(a): source policy deviation
C1 R ap
Drv(PMan MP
constant.
PM Daro
(c): dynamics mismatch
The above bound depicts that the performance deviation of a policy in the true target domain and the empirical source domain determined by three factors: term (a) and (b) that measure the deviation between the learned policy and the behavior policy in the source domain dataset and the target domain dataset, respectively; term (c) that measures the dynamics mismatch between the target domain dynamics and the empirical transition dynamics in the source domain dataset.
Remark:
3.2 A NOVEL OBJECTIVE
Theorem 3.1 conveys that a promising way to improve the performance of the learned policy in the target domain when leveraging source domain offline data is to minimize the dynamics mismatch
3

---

Published as a conference paper at ICLR 2025
between the target domain and the empirical source domain, i.e., Drv PP Naturally, this can be achieved by only keeping source domain data that have similar transition dynamics as the target domain dynamics. Motivated by this insight, we can formulate the objective function for training the value function Qo a) parameterized by 0 as:
L = EE [1Qa TQp) + E((s,a.")~DDer [1 <) TQp) + A0 TQn), (2)
where 1-) is the indicator function, p measures the probability that the sampled source domain transition lies within the span of the target domain dataset, e threshold hyperparameter, 7 the Bellman operator, and Rp... (Qe. TQe) denotes the regularization term on source domain data. In practice, directly optimizing Equation 2 not preferred since one may need manually determine and adjust the regularization term per dataset. Instead, we propose reject fixed proposition of data in the sampled batch and combine the regularization term with the Bellman error to attain an implicil regularization. Formally, the objective function for the value function can be formulated as:
f== = Ep,.. [[Q. TQ + E's,a,"")>D... [wrss o, 1') > Pex - TQo)
(3)
where Pe% is the top E-quantile likelihood estimation of the sampled batch from the source domain dataset, w(s, a, 5²) is the weight that related to p. i.e. w(s, a, 8') is large when p(s, a, s') is large (i.e., (s, a, s') lies close to the target domain) and vice versa. Equation 3 is appealing because it filters source domain data that deviate far from the target domain and adaptively penalizes the rest samples. Then, it remains to decide how to empirically calculate ñ and w(s, a, s'). We notice that measuring á can be equivalent to measuring the deviation of the source domain sample against the target domain. Some studies estimate the dynamics gap by training domain classifiers (Liu al., 2022a), performing contrastive learning (Wen et al., 2024), etc. However, they often involve training neural networks, which can be inferior given few target domain data due overfitting. Instead, we propose to adopt the optimal transport (OT) approach, which is a principled method for comparing two distributions, to align data the source domain dataset and the target domain dataset.
We define u = Sarc a asre O S'rre and u' Star a Grar i starr where ① is the the vector concatenation Parel
operator, (Sarc, Carc: s'ree) Daro; (Star, Gtar, s(ax) Diar- Let Ps [Drre] DlDrel 6u, and Pr
Diae Elera dug denote the tate-action-neex--satee joint distribution of the source domain dataset
and the target domain dataset, respectively. Given a cost function C, the Wasserstein distance
D Diac
W(u, min PEM
Σ
Σ
Clun, )
(4)
f=1
can depict the distance between datasets from two domains. Suppose the optimal coupling by solv- ing the optimization problem in Equation 4 gives μ², we determine the deviation between source domain data and the target domain dataset via:
Deael d(u,)= = Σ
Clur, dh = (S% dgress (8'sre))) ~ Dare.
(5)
Intuitively, d is larger if the source domain data aligns the distribution of the target domain dataset (since the cost is smaller by then) and smaller otherwise. It can hence work as a good proxy for in Equation 3. Furthermore, we show in Theorem 3.2 that d serves as an upper bound of the negative total variation deviation between two empirical distributions Ps, Pt-
Theorem 3.2. Assume that the cost is bounded, i.e., Clu, 2²)= < Cmax < 00, V u, u', then we have
0 ž d(wu) 2 -Cass (p=llpi)
The assumption can be easily satisfied with an appropriate cost function (e.g., Euclidean distance) since the state space and action space usually bounded. The above theorem further validates the rationality of using d where it bounds the dynamics discrepancy between two offline datasets.
Another benefit of computing is that provides a natural way of measuring the weight w(s, a, s') in Equation 3, e.g., w(s, a, s') ) = exp(a x d(u)), where a >> 0 is a hyperparameter. This is valid because E [0, 1] given d(u) < 0. Consequently, the objective function becomes
LQ ED.... [Qov T + E(e.,,'') [exp(a dd > dex) (Qn TTQ)) (6)
4

---

Published as a conference paper at ICLR 2025
where we abbreviate d(u) d(s, s') ) and de% denotes the top E-quantile likelihood estimation of the sampled source domain batch. However, is insufficient to merely optimize Equation 6 since the performance bound in Theorem 3.1 is also connected with policy deviation terms. If only limited target domain data is available (e.g., 5000 transitions), the learned policy can get biased towards the behavior policy of the source domain dataset, incurring unsatisfying policy adaptation performance.
As remedy, we include an extra policy regularization term that encourages the learned policy to be close to the support region of the target domain dataset (namely, dataset regularization), Let Cx be the vanilla policy optimization objective of the underlying offline RL algorithm, we modify it into
E La - 3 x Bew log 6 (sT{|s)]ss,,
where B > 0 is the weight coefficient, a is the behavior policy in the target domain dataset. Then, minimizing Ex guarantees that the probability of the learned policy lying the span of the target domain dataset is maximized. In this way, term (b) in Theorem 3.1 can be better controlled. Term (a) can also be minimized since the agent trains upon the source domain data.
Formally, we introduce our novel Optimal Transport Data Filtering (tagged OTDF) approach with its framework outlined in Figure 1, which is built upon the objective functions in Equation 6 and Equa- tion 7. In principle, our framework can be integrated into any off-the-shelf offline RL algorithms. The abstracted pseudocode of OTDF is presented in Algorithm
Algorithm 1 Optimal Transport Data Filtering (Abstracted Version)
Input: Source domain dataset Dare. target domain dataset Dear, batch size N, data selection ratio
1: Initialize policy Tg. value function Qe. the cost function C, coefficients a, ,3
2: Compute the optimal alignment between Dare and Dtar with Equation 4
// Solve OT
3: Compute deviations {de) |Daell between the source domain data and Drar with Equation 5 [Darel
4: Concatenate Dare and (d, Deral to get Derc = {88, (1, T1, 8/, (d)))]e
5: f 1, 2, ...do do
N from Dave 6: Sample mini-batch bare == {(s(((, a,,,,, d)} with size 2
7: Sample mini-batch bear == {(s, a. s')) with size 2/ from Dvar
8: Optimize the value function Qo on bare U bear with Equation
I/ Data filtering Dataset regularization
9: Optimize the policy πe on lare btar with Equation
10: end for
3.3 PRACTICAL ALGORITHM
Intuitively, OTDF involves two crucial novel components: (a) selective source domain data sharing via optimal transport alignment, and (b) policy constraint to align with the target domain dataset.
For the first component, one can solve the OT problem in Equation 4 before initializing the offline RL algorithm, during the policy optimization iterations. In practice, resort to the former to save time used in solving the repetitive OT problem under different seeds, as shown in Algorithm 1. To obtain the optimal coupling matrix μ in Equation 4, we solve the entropy-regularized OT problem with Sinkhorn's 's algorithm (Cuturi, 2013), using the Sinkhorn solver in OTT-JAX (Cuturi et al., 2022). The OTT-JAX library provides highly efficient and scalable implementation of the Sinkhorn algorithm using GPUs. As a result, we can compute the deviations in Equation 5 for million source domain transitions within five minutes, given about 5000 target domain data.
We then can leverage Equation 6 to train the value function. However, the resulting deviations can be largely affected by the underlying task (e.g., different tasks have varied state spaces and action spaces) and dataset qualities (e.g., d obtained from the medium-level source domain dataset and the expert-level source domain dataset can differ, given the same target domain dataset). This indicates that one may manually decide the coefficient a per dataset to acquire suitable weights, which hinders the practical application of OTDF. To mitigate this concern, we propose to normalize the deviations:
d, - max; d; max; d; - min; ' E (1,2,..., N).
d.
where is the size of the sampled source domain batch. We subtract max, d, to ensure that d, lies in the range of []. We find this min-max normalization approach quite effective, diminishing
5

---

Published as a conference paper at ICLR 2025
the need for the extra hyperparameter a. We therefore optimize the following objective alternatively:
L .... [[Qo TQ&)²] ( exp(())(( dex)(Q0 - - TQo)3
(9)
For the second component, it necessities to model the behavior policy of the target domain dataset. We fulfill this by using the conditional variational auto-encoder (CVAE) (Kingma & Welling, 2013), which we find can model the behavior policy well even under a limited budget of data. The CVAE G consists of an encoder E, (s, a) and a decoder D. (s. 2) parameterized by v, s respectively. The objective function for training the CVAE in the target domain dataset gives,
LCVAE (xx [(a (s, + DKL (Ev(s,a))
(10)
where DKL (pllq) denotes the KL-divergence between two probability distributions p. 9, and I the identity matrix. We pretrain the CVAE policy upon the target domain dataset before training OTDF. Given a state sampled from the mixed dataset Dare U Dtar: we sample the corresponding action from the learned policy ~ rT-(s), and feed them into the encoder E to produce M latent variables z. Afterward, we input and z into the decoder D to reconstruct actions ā that come from the same distribution the target domain dataset and construct Gaussian distributions based them. We then compute the log probability of a (from т) belonging to these Gaussian distributions. Finally, approximate Equation 7 by measuring the logsumexp of the log probabilities:
M
(
= Bx
EE ard DareDtuuDD UDar
log
exp(log Fla (T(()))s)))
(11)
where 7la denotes the i-th constructed Gaussian distribution with the decoded ā (based on 2, i E M)) as the mean and constant as the standard deviation. We find that OTDF is robust to different choices of M. We hence set M = 10 by default and do not bother tuning it. Apparently, Equation 9 and Equation 11 not alter the vanilla objectives of the base offline RI. methods and can serve as an add-on module for them. In this work, we build the practical OTDF algorithm upon IQL (Kostrikov et al., 2022) and defer the detailed pseudocode of OTDF+IQL to Appendix D.2.
4 EXPERIMENTS
In this section, we examine the effectiveness of our proposed method by conducting experiments on environments with various dynamics shifts. We aim to answer two questions: (a) Can OTDF fulfill effective offline policy adaptation and boost the performance of the base method? (b) Can OTDF beat prior strong baselines across varied dynamics shifts and dataset qualities? We further present detailed parameter study to promote better understanding of OTDF.
4.1
MAIN RESULTT
Tasks and datasets. To comprehensively evaluate the policy adaptation capabilities, we consider three kinds of dynamics shifts, including gravity shift, kinematic shift, and morphology shift, for four environments (halfcheetah, hopper, walker2d, ant) from OpenAl Gym (Brockman et al., 2016). The gravity shifts are realized by modifying the strength of the gravity (the direction of the gravi- tational force remains unchanged). We simulate the kinematic shifts by limiting the rotation range of some joints, and the morphology shifts by modifying the size of some limbs. More details on the environment settings can be found in Appendix C. Since we consider the cross-domain RL set- ting where only limited target domain data can be accessed, we use D4RL (Fu et al.. 2020) "-v2" MuloCo datasets as the source domain datasets and manually gather offline datasets in those mod- ified environments to serve as target domain datasets, which contain only around 5000 transitions. This poses considerable challenges for existing offline RL methods to achieve good performance when merely training on the target domain, as observed in Liu et al. (2024a); Wen et al. (2024). We follow similar data collection procedure D4RL to collect medium, medium-expers and expert target domain datasets. We allow source domain dataset qualities to be medium, medium-replay and medium-evee expert. This amounts to total of 36 tasks for single shift.
Metrics. We are primarily interested in the performance of the agent in the target domain. Since the scales of the return can differ in varied environments, we follow D4RL and evaluate the normalized

---

Published as a conference paper at ICLR 2025
score metric: NS- - -- x 100, where J, Je. J,, denote the return of the leaned policy, the expert policy and the random policy, respectively. NS 100 corresponds to an expert policy and NS = 0 corresponds to a random policy. Please see Appendix C.1 for more details on the datasets.
Baselines. We consider the following baselines: IQL (Kostrikov et al., 2022) that train the IQL policy upon both source domain data and target domain data; DARA (Liu et al., 2022a) that esti- mates the dynamics gap by training domain classifiers and utilizes it for penalizing source domain rewards; BOSA (Liu et al., 2024a) that employs support-constrannd objectives to regularize the value function and the policy from being OOD; SRPO (Xue al., 2024) that leverages the station- ary state distribution as regularizer for reward modification; IGDF (Wen et al., 2024) that filters source domain data by introducing contrastive learning objective. The implementation details of these baselines can be found in Appendix D.I.
Results. We run all algorithms for 1M gradient steps across 5 random seeds. We summarize perfor- mance comparison results of OTDF against baselines under morphology shifts and gravity shifts in Table 1 and Table 2, respectively. Due to space limits, the empirical results under the kinematic shifts are deferred to Appendix 1. We report the normalized score performance in the target domain.
Answering question (a): Based on the empirical results, it is evident that OTDF significantly out- performs IQL* in most scenarios. Notably, OTDF achieves higher normalized scores than IQL* on 27 out of 36 tasks under the morphology shifts and 29 out of 36 tasks under the gravity shifts. IQL* on the remaining tasks. Adopting OTDF incurs an increase of 59.7% and 40.7% % in terms of the total normalized score under the morphology and gravity shift tasks, respectively. These clearly validate the effectiveness of our method.
OTDF exhibits competitive performance
Answering question (b): We find that OTDF significantly surpasses recent baselines across numer- ous dataset qualities and types of the dynamics shift scenarios, often by a large margin. Specifically, OTDF excels in 23 out of 36 tasks (with varied dataset qualities of both domains) under the mor- phology shifts and achieves a total normalized score 1274.3, while the second best baseline (DARA) only has a total score of 816.8. Under the gravity shift tasks, OTDF markedly beats other methods across 26 out of 36 tasks, exceeding the second best approach (DARA) by 36.4% in terms of the total normalized score performance. Despite OTDF's suboptimal performance in some tasks, it still remains competitive compared to other methods in those tasks.
We observe that the most recent basclines have similar performance as IQL on many tasks, indicat- ing that they fail to fulfill effective offline policy adaptation. OTDF is the only method that exhibits remarkable performance gain compared to the base method. This can be possibly attributed to the fact that it is difficult to learn classifiers (DARA, SRPO) or dynamics transition model (BOSA) given a limited budget of the target domain dataset. The reward penalties provided by DARA and SRPO can hence be poor, which ultimately results in negative effects on policy training. Furthermore, the policy trained by BOSA can be biased and favor the distribution of the source domain dataset instead of the target domain dataset, because BOSA constrains the learned policy to stay within the support region of the mixed dataset De Drar. OTDF lifts these concerns by selectively sharing source do- main data with OT (no neural network training) and enforcing target domain dataset regularization. Since the computation of OT is not affected by the coverage or the quality of the adopted datasets, OTDF can consistently keep its advantages over baselines across numerous tasks regardless of the dataset qualities from both domains and types of dynamics shifts.
4.2 PARAMETER STUDY
In this part, we investigate how sensitive OTDF is to the introduced hyperparameters. There are two crucial hyperparameters in OTDF, the data selection ratio & and the policy coefficient B.
Data selection ratio &. E decides how many source domain data in sampled batch can be shared for policy training. A larger indicates that more source domain data will be admitted. To examine its influence, we conduct experiments with medium source domain datasets and medium target domain datasets and sweep & across (0, 20, 40, 60, 80, 100). E = 100 means that no data selection process is included in OTDF while & = 0 means that all source domain data are rejected when learning the value function. We summarize the experimental results in Figure 2 and observe that it is not ideal to set { = or 100. This ablates the necessity of the data filtering process with OT. Note that different
We add * IQL to highlight that it is trained on the mixed dataset.
7

---

Published as a conference paper at ICLR 2025
Table Performance comparison in cross-domain offfine RL given the morphology shift. half= halfcheetah, hopp = hopper, walk walker2d, m medium, r= repla,, e expert. The target column denotes the offline dataset quality the target domain. We report normalized scores and standard deviations in the target domain under varied dataset qualities of the source domain data (medium, medium-replay, medium-expert and target domain data (medium, medium-expert, expert). The results are averaged over 5 seeds. We bold and highlight the best cell.
Source
Target
IQL*
DARA
BOSA
SRPO
IGDF
OTDF (ours)
half-m
medium
30.0+1.6
26.61:33
19.3:3.5
41.310..4
39.1=2.3 41.6+1.55
half-m
medium-expert
31.811.1
32.010.7
33.611.1
30.710.88
29.6122.
35.610.7
half-m
expert
8.5=100
9.311.6
7.900.8
8.610.9
10.0.10.8
10.7:11.2
half-m-r
medium
30.8++4.4
35.6+0.7
35.0+44.6
32.0+114
28.0++2.0
40.0++1.2
half-m-r
medium-expert
12.94222
16.944.1
19.945.55
12,4+1.6
12.0+3.7
34.4+00.7
half-m-r
expert
5.9-1.7
3.7+2.7
2.4+1.9
6.2+1.4
5.3+2.3
8.2+22.7
half-m-c
medium
41.5++0.1
40.3+1.2
41.3++0.3
41.3++0.4
40.9+10.4
41.4+-0.3
half-m-e
medium-expert
25.8.2..0
30.6.1..8
32.110.88
27.210.88
26.2.11.8
35.1
half-m-e
expert
7.8-1.3
8.3.1.3
9.110.8
7.810.9
7.510.9
9.8.1.0
hopp-m
medium
13.5.10.2
13.5
13.210.3
13.4.0.1
13.410.2
11.0.1.9
hopp-m
medium-expert
13.4
13.6=0.2
11.244.6
13.3+0.2
13.3+0.4
12.6+0.8
hopp-m
expert
13.5+02
13.6#0.3
13.3+0.4
13.6
13.940.1
10.7+4.7
hopp-m-r
medium
10.8+1.1
10.2=1.0
1,2+0.0
10,7#1.6
12.0
8.742.8
hopp-m-r
medium-expert
11.6+11.6
10.4
1.340.22
10.411.2
8.24-2.8
9.7+-2.7
hopp-m-r
expert
9.8-0.5
9.0+-..3
1.340.1
10.4+:.4
11,41
10.7+:2.4
hopp-m-e
medium
12.6+:1.4
13.0+0.5
15.747.2
14.0+-2.3
12.740.8
7.9+3.22
hopp-m-e
eedium-expert
14.1
13.820.6
12.0114
13.510.3
13.311.2
9.6:3..5
hopp-m-e
expert
13.810.5
12.311.8
10.545.0
14.742.3
12.8+0.9
5.914.0
walk-m
medium
23.01:4.7
23.3133
6.242.9
24.7.1..7
27.519.5
50.515.8
walk-m
medium-expert
21.548.6
22.247.6
7.2+2.9
18.747.3
20.7+5.99
44.3++23.8
walk-m
expert
20.3+22.
17.3+3.4
15.8+8.7
21.147.2
15.8+4.5
55.3+-8.3
walk-m-r
medium
11.3+3300
10.9+4.6
5.444.0
10.444.8
13.4+7.2
37.4+5.1
walk-m-r
medium-expert
7.0-1.5
4.511.11
4.0.1..2
4.911.7
6.9122.2
33.8
walk-m-r
expert
6.3-0.9
4.5.1.11
3.8.3..4
5.510.9
S.5.12.2
41.5
walk-m-e
medium
24.111..4
31.716.6
18.716.5
29.914.7
27.5:1..3
49.9.14.6
walk-m-e
medium-expert
27.0:15.5
23.315.5
11.140.9
22,913.8
25.3:6.4
40.5
walk-m-e
expert
22.41t333
25.2:5.77
9.93399
18.7+5.7
24.7+22.4
45.7=16.9
ant-m
medium
38.7=13.8
41.3
18.2+19
40.6+2.1
40.9+1.7
39.4+1.7
medium-expert
47.0+t5.1
43.3+2.0
45.3+7.0
47.2+4.3
44.4+11.
58.3+-8.9
ant-m expert
36.24:3.5
48.5+4.2
12.2+-0.5
42.24-9.9
41.4+4.22
85.4+-4.4
ant-m-r medium
38.24-2.9
38.9+2.7
20.243.7
38.3+-199
39.7+1.2
41,2+-0.9
ant-m-r
medium-expert
38.1
33.41:5.5
15.2
35.015.7
37.312.4
50,8
ant-m-r
expert
24.11:1.9
24.512.6
16.0.1..7
22.71:3.0
23.6.11.4
67.2.17.5
ant-m-e medium
32.915.11
40.2.11.5
28.115.66
35,9122.5
36.114.44
39,9112,9
ant-m-e
medium-expert
35.74:3.9
36.5+88.7
14.8+15.9
24.5+=1.7
30.7+
65.7+4..5
ant-m-e
expert
36.1
34.6+t5.8
53.9+5.00
38,4+9.4
35.2+6.6
86,4+2222
Total Score
798.0
816.8
646.3
803.1
808.7
1274.3
8-
20
E- 40
E- 60
E- 80
E- 100
hallchootah- Knneatteeeee
walker2d knnmmaico
hatfcheetah-maan
walker2d marph
20th
Return
2000
1590
3000
1500
3900
190
699
2700
500
2.4
200g
32
CA
8.2
Milion Steps
Milion S
Millonn Steps
Milioo Steps
Figure 2: Parameter study of the data selection ratio &. *-kinematic denotes the kinematic shift task, while "-morph denotes the morphology shift tasks. We use halfcheetah, walker2d tasks. The solid lines depict the average returns seeds and the shaded area denotes the standard deviation.
8

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison under the gravity shift. half halfcheetahh hopp hopper,
walk = walker2d, m = medium, r= replay, expert. The target column denotes the offline dataset quality of the target domain, The normalized average scores in the target domain across 5 random seeds are reported and + captures the standard deviation. We bold and highlight the best cell.
Source
Target
IQL*
DARA
BOSA
SRPO
IGDF
OTDF (ours)
half-m
medium
39.6+33.3
41.2++3.9
38.9+-4.0
36.9+-4.5
36.6+5.5
40.7+7.77
half-m
medium-cxpert
39.6+:3.7
40.7+2.8
40.4+3.0
40.7+-2.3
38.7+6.2
28.6+3.2
half-m
expert
42.4++3.8
39.8+4.4
40.5+3.9
39.4+1.6
39.6+4.6
36.1+5.3
half-m-r
medium
20.11:5.0
17.616.2
20.014.9
17.51:5.2
14.4112.2
21.5.16.5
half-m-r
medium-expert
17.2.1..66
20.2.15.2
16.7144.2
16.31117
10.0.2..5
14.7.4.11
half-m-r
expert
20.7.15.5
22.411.77
15,414.2
23.11.4.0
15.313.77
11.4.1199
half-m-e
medium
38.6+6.0
37.8+333
41.8+5.1
42.5+t2.3
37.7=7.3
39.5+33.5
half-m-c
mepert
39.6+3.0
39,4+4.4
38.7+3.7
43.3+2.7
40.7#3.2
32.4=5.5
half-m-e
expert
43.4+0.9
45.3+113
39.9+2.7
43.3+3.0
41.1#4.1
26,5
hopp-m
medium
11.2+:1.1
17.343.8
15.2433.
12.4+1.0
15.3+3.5
32,4+8.0
hopp-m
medium-expert
14.7::3.6
15.4+22.5
21.149..
14.2+11.8
15.143.6
24.2++3.6
hopp-m expert
12.5+t1.6
19.3+
12.7+1.7
11.8+0.9
14.8+4.0
33.7+7.88
hopp-m-r medium
13.9:22.9
10.744.3
3.311.9
14.042.6
15.314.4
31.1=11.44
hopp-m-r
medium-expert
13.3:16.3
12.545.6
4.611.77
14.414.2
15.415.5
24.2:16.1
hopp-m-r
expert
11.0::2.6
14.316.0
3.210.8
16.415.0
16.114.0
31.0-9.8
hopp-m-e
medium
19.1+66.6
18.5+12.3
15.9+5.9
19.7+8.5
22.345.4
10.4+10.1
hopp-m-c
medium-expert
16.8+2.77
16.046.1
17.3+2.5
15.8+3.3
16.6+7.7
28.3+6.7
hopp-m-c
expert
20.9+t.11
23.9+14.8
23.247.9
21.41.9
26.0=9.2
10.9
walk-m
medium
28.1111.9
28.4113.7
38.011112
21.477.0
22.118.4
36.612.3
walk-m
medium-expert
35.7114.7
30.719..7
40.917.2
34.01999
35.419.1
44.8
walk-m
expert
37.3118.
36.017.00
41.318.6
39.513.8
36.2.13.6
44.0
walk-m-r
medium
14.6:12.5
14.116.1
7.6.1.88
17.91388
11.614.6
32.7-7.00
walk-m-1 m-r
medium-expoo expert
15.3+1.9
15,945.8
4.8:5.88
15.3+4.5
13.9
31.6=t.11
walk- walk-m-r
expert
15.8+7.2
15.7+4.5
7.144.6
13.7+8.1
15.245.3
31.3
walk- m-e
medium
39.9+13.11
41.6+13.0
32.347.2
46.4+3.5
33.823.1
30.2-19.8
walk-m-e
medium-expert
49.1+6.9
45.8+9.4
40.1+4.5
36.4+3.4
44.7+229
53.3+7.1
walk-mme
expert
40,4+11.9
56.443.5
43.7+4.4
45.8+8.0
45.3+10.4
61.1+3.4
ant-m medium
10.2+1.8
9.4+0.9
12.4+2.0
11.7+1.0
11.3+
45.1+12.4
ant-m
medium-expert
9.411.22
10.010.9
11.6113
10.21112
9.411.4
33,915.4
ant-m expert
10.2:10.3
9.810.66
11.810.4
9.510.6
9.711.
33.2.19.0
ant-m-r medium
18.91:2.6
21.712.1
13.911.5
18.711.7
19.6.1.0
29.6
ant-m-r
medium-expert
19.1+3.0
18.3+2.1
15.9+2.7
18.7+1.8
20.3
25.4+2.1
ant-m-r expert
18.5+009
20.0+1.3
14.5+1.7
19.9+2.1
18.8+2.1
24.5
ant-m-e medium 9.8+2.4
8.111.8
8.1+3.0
8.4+2.1
8.9+1.5
18.6+11.9
ant-m-e
medium-expert
9.0-10.8
6.4.1.44
6.2.1..5
6.1.3..5
7.2.:2.9
34.0
ant-m-e
expert
9.14:2.6
10.4-2299
4.2.1.99
8.8.11.0
9.2.11.5
23.2.12.9
Total Score
825.0
851.0
763.2
825.5
803.6
1160.7
tasks prefer varied & while we can achieve trade-off with & = 80. 80% source domain data) for OTDF by default and do not it.
We hence set ç = 80 (i.e., share
8
8 0.5
2.0
ant-tinemttic
hooper-kinemaiis
ani-morph
hopper-morph
1500
Rolum
100
3000
2000
200
1000
Milion Steps
Millon Steps
Millin Steps
Millon Steps
Figure 3: Parameter study of the policy regularization coefficient 3. We report the returns ob-
tained in the modified hopper and ant tasks. The shaded region captures the standard deviation.
Policy coefficient B. 3 controls the strengths of the target domain constraint term Equation 11. A larger 3 may de-emphasize the knowledge carried by the source domain data while small B can

---

Published as a conference paper at ICLR 2025
incur biased policy that leans towards the distribution of the source domain dataset. Obviously, 3 needs to be set properly to achieve a trade-off. We employ 8 E {0, 0.5, and run OTDF on the meppam-replay source domain datasets and expert target domain datasets. We present the results in Figure 3 and find that diminishing the role of dataset regularization (i.e., B 0) leads to unsatisfying performance. OTDF can be sensitive to the choice of 3 and the best /3 for each task vary (e.g., ant-kinematic prefers 3 = 2.0 while hopper-kinematic favors 3 = 0.5). Fortunately, we can reach a trade-off with [ 0.5 or 3 0.1. For most of our experiments, we set B 0.5.
RELATED WORK
Offline Reinforcement Learning (RL). In Offline RL (or batch RL) (Levine et al., 2020; Lange et al., 2012), the agent is only allowed to learn policies based on some previously gathered offline datasets. Offline RL methods can be generally divided into model-free (Wu et al., 2021; An al., 2021) and model- based (Argenson & Dulac-Aroold, 2020; Matsushima et al., 2021) approaches. The success of these offline RL methods relies heavily on the fact that the underlying static datasets usually contain large amount of transitions. Instead, we investigate how to facilitate target domain policy training by leveraging source domain datasets, given only few target domain data.
Domain Adaptation in RL. In this work, we focus on the policy adaptation problem (Xu et al., 2023; Lyu et al., 2024a;b) under dynamics shifts between the two domains, while other components like observation spaces unchanged. To mitigate this issue, previous methods mainly leverages domain randomization (Slaoui et al., 2019; Mehta et al., 2019), system identification (Clavera et al., 2018; Du al., 2021), imitation learning (Kim et al., 2019; Hejna al., 2020), and meta-RI. meth- ods (Nagabandi et al., 2018; Raileanu et al., 2020), etc. However, they often require expert demon- strations from the target domain, and prior knowledge to guide parameter randomization. Some recent works discard these demands and train dynamics-aware policies given limited offline transi- tions from one domain while having access to another online domain with dynamics discrepancies (Niu et al., 2022; Xu et al., 2023; Lyu et al., 2024a). In contrast, we explore the offline policy adaptation setting where the two domains are both offline. Existing works address this issue via performing reward penalization (Liu et al., 2022a), conducting data filtering from the perspective of mutual information (Wen et al., 2024), etc. Different from these methods, we resort to the opti- mal transport approach for selecting source domain transitions that are close to the target domain, in conjunction with a policy constraint term that enforces the learned policy to stay close to the support region of the target domain. These allow our method to function well even under very few target domain samples where existing methods typically fail.
Optimal Transport (OT). OT is widely used in domain adaptation (Damodaran et al., 2018; Shen et al., 2018), recommender system (Li et al., 2019; Mashayekhi et al., 2023), and graph matching (Chen et al., 2020; Xu et al., 2019). In the context of RI, OT is used in fields like imitation learning (Haldar et al., 2023; Nguyen et al., 2021; Dadashi et al., 2020; Luo et al., 2023), curriculum RI. (Klink et al., 2022; Huang et al., 2022), preference- bassed RL (Liu et al., 2024b), etc. We, instead, resort to OT for data filtering in the context of cross-domain offline RL.
6 CONCLUSION
In this paper, we study the cross-domain offline policy adaptation problem which seeks to enhance offline policy training on limited target domain dataset by leveraging a source domain dataset with dynamics shifts. We theoretically characterize the performance bound of a policy under this setting, which further motivates us to perform data filtering upon source domain data with optimal transport and introduce dataset regularization term to maximize the probability that the learned policy stays within the span of the target domain dataset. These give birth to the OTDF algorithm, which compatible with any offline RL methods. Empirically, we combine OTDF with IQL and evaluate its performance upon datasets with distinct qualities and dynamics shift types. Experimental results demonstrate that OTDF significantly exceeds baselines numerous tasks, often by large margin.
The limitations of our work lie in that: (a) OTDF is inapplicable when the source domain and the target domain have varied state spaces action spaces; (b) our experiments only carried out in the simulated environments, and remains to explore the effectiveness of OTDF in worldworld scenarios. We plan to resolve these drawbacks in future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the STI 2030-Major Projects under Grant 20212D0201 404 and NSFC under Grant 62450001. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline re- inforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. arXiv preprint hll::0008.05556, 2020.
Karol Arndt, Murtaza Hazara, Ali Ghadirzadeh, and Ville Kyrki. Meta reinforcement learning for sim-to-real domain adaptation. In JEEE International Conference on Robotics and Automation, 2019.
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakr- ishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, and Vincent Van- houcke. Using simulation and domain adaptation to improve efficiency deep robotic grasping. In IEEE International Conference Robotics and Automation, 2018.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. ArXiv, abs//600..01 01540, 2016.
Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. In International Conference on Machine Learning, pp. 1542-1553. PMLR, 2020.
Ignasi Clavera, Anusha Nagabandi, Ronald S. Fearing, Peter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt: Meta-learning for model-basee control. ArXiv, abss1803. 11347, 2018.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman, Quantifying generaliza- tion in reinforcement learning. In International conference on machine learning, pp. 1282-1288 1289.
PMLR, 2019.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Marco Cuturi, Laetitia SaangPaaaxanthos, Yingtao Tian, Charlotte Bunne, Geoff Davis, and Olivier Teboul. Optimal transport tools (ott): A jax toolbox for all things wasserstein. arXiv preprint arXiv:2201. 12324, 2022.
Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imita- tion learning. arXiv preprint arXiv:2006.044: 2020.
Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European conference on computer vision (ECCV), pp. 447-463, 2018.
Yuqing Du, Olivia Watkins, Trevor Darrell, P. Abbeel, and Deepak Pathak. Auto-tuned sim-to-real transfer. IEEE International Conference on Robotics and Automation, 2021.
Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhut- dinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In
International Conference on Learning Representations, 2021. URL net/ iitoorummiideee weeiiiieeeessssa IACIA.
Arnaud Fickinger, Samuel Cohen, Stuart Russell, and Brandon Amos. Cross-domain imitation
learning via optimal transport. In International Conference on Learning Representations, 2022.
URL https: petttps://ppereeeeee rwwpeeeeeeeeeeeeeeeeeeeeee net t

---

Published as a conference paper at ICLR 2025
Justin Fu, Aviral Kumar, Ofir Nachum, G. Tucker, and Sergey Levine. D4RL: Datasets for Deep
Data-Driven Reinforcement Learning. ArXiv, abs/2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A Minimalist Approach to Offline Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-Policy Deep Reinforcement Learning without Exploration. In International Conference on Machine Learning (ICMI.) 2019.
Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via amage-to-image translation. In International Conference on Machine Learning, 2018.
Yuying Ge, Annabella Macaluso, Erran L.. Li, Ping Luo, and Xiaolong Wang. Policy adaptation from foundation model feedback. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Kaiyang Guo, Shao Yunfeng, and Yanhui Geng. Model-based offline reinforcement learning with pessimisn-modulatedd dynamics belief. Advances in Neural Information Processing Systems, 35: 449-461, 2022.
Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, and Anqi Liu. Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation. In The Thirty-eighuh Annual Conference on Neural Information Processing Systems, 2024. https: wpwpeeeeeeeeee..eeee/ee net/Eorum? der22id--a k2hS5Rt 1NO.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft Actor-Critic Algorithms and Applications. arXiv preprint arXiv:/812.059005 2018.
Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Lerrel Pinto. Watch and match: Supercharg- ing imitation with regularized optimal transport. In Conference on Robot Learning, pp. 32-43. PMLR, 2023.
Donald Joseph Hejna, P. Abbeel, and Lerrel Pinto. Hierarchically decoupled imitation for morpho- logical transfer. ArXiv, abs/2003.01709, 2020.
Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum rein- forcement learning using optimal transport via gradual domain adaptation. Advances in Neural Information Processing Systems, 35:10656-10670, 2022.
Yuan Jiang, Chenglin Li, Wenrui Dai, Junni Zou, and Hongkai Xiong. Variance reduced domain randomization for reinforcement learning with policy gradient. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16:1031-, 2023.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- Based Offline Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Kuno Kim, Yihong Gu, Jiaming Song. Shengjia Zhao, and Stefano Ermon. Domain adaptive imita- tion learning. In International Conference on Machine Learning, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Inter- national Conference on Learning Representation, 2015. URL http://arxiv.or abs/ 1412. 6980.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes, ArXiv, abs/1312.6114 2013.
Pascal Klink, Haoyi Yang, Carlo D' D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforce- ment learning via constrained optimal transport. In International Conference on Machine Learn- ing, pp. 1 11341-11358. PMLR, 2022.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022. URL https:/ /// openreview net/ 6= 9ZJWF8.
12

---

Published as a conference paper at ICLR 2025
Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine Conservative Q-Learning for Offline Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch Reinforcement Learning. In Rein- forcement Learning, 2012.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. ArXiv, abs/2005.01643, 2020.
Ruilin Li, Xiaojing Ye, Haomin Zhou, and Hongyuan Zha. Learning match via inverse optimal transport. Journal of machine learning research, 20(80):1-37, 2019.
Jinxin Liu, Zhang Hongyin, and Donglin Wang. DARA: Dynamics-aware reward augmentation in
offline reinforcement learning. International Conference on Learning Representations, 2022a. URL https::///// /ttp:::://peeeee /wwp///eeeeeeeeeeeeeeeeeeeeeeeee net forum? -ed-idd--5i 3d-
Jinxin Liu, Ziqi Zhang, Zhenyu Wei, Zifeng Zhuang, Yachen Kang, Sibo Gai, and Donglin Wang. Beyond ood state actions: Supported cross-domain offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 13945 13953, 2024a.
Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, and Xiu Li. Pearl: Zero-shot cross-task preference alignment and robust reward learning for robotic manipulation. In International Conference on Machine Learning, 2024b.
Xingyu Liu, Deepak Pathak, and Kris M. Kitani. Revolver: Continuous evolutionary models for robot-to-robot policy transfer. International Conference on Machine Learning, 2022b.
Wang Luo, Haoran Li, Zicheng Zhang, Congying Han, Jiayu Lv, and Tiande Guo. Sambo-rl: Shifts- aware model-based offline reinforcement learning. arXiv preprint arXiv:2408. /2830, 2024.
Yicheng Luo, Zhengyao Jiang, Samuel Cohen, Edward Grefenstette, and Marc Peter Deisenroth.
Optimal transport for offline imitation learning. arXiv preprint arXiv:2303. 13971, 2023.
Jiafei Lyu, aicheng Gong, Le Wan, Zongqing Lu, and Xiu Li. State advantage weighting for offline RL. In 3rd Offline RL Workshop: Offline RL as "Launchpad", 2022a. URL https:// openreview. net/ forum? dartumii==rrroou
Jiafei Lyu, Xiu Li, and Zongqing Lu. Double check your state before trusting it: Confidence-aware bidirectional offline model-based imagination. In Advances Neural Information Processing Systems, 2022b. URL https: attps:///peeneviiee wppprevieeeeeeeeeeee/ee/ee net/ Eorumrirddd -edd-edddd3 3IIMLDSL.
Jiafei Lyu. Xiaoteng Ma. Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In Neural Information Processing Systems, 2022c
Jiafei Lyu, Chenjia Bai, Jingwen Yang. Zongqing Lu, and Xiu Li. by capturing representation mismatch. arXiv preprint 2rXiv:2405. 15369, 2024a.
policy adaptation
Jiafei Lyu, Kang Xu, Jiacheng Xu, Mengbei Yan, Jing- Wen Yang. Zongzhang Zhang, Chenjia Bai, Zongqing Lu, and Xiu Li. ODRL: benchmark for off-dynamics reinforcement learning. The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024b. URL https: httpttttttpteeeeeeeeeeeee wptpp//wnwee...ccoooeeee net/ htt////oommmmaaa/amomo 4x1xkxrryy...
Yoosof Mashayekhi. Bo Kang, Jefrey Lijffijt, and Tijl De Bie. Recon: Reducing congestion in job recommendation using optimal transport. Proceedings of the 17th ACM Conference on Recommender Systems, pp. 696-701, 2023.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- Efficient Reinforcement Learning via Model-Basse Offline Optimization. In International Con- ference on Learning Representations (ICLR), 2021.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher Joseph Pal, and Liam Paull. Active domain randomization. ArXiy, abs/ Abs/19944 2019.
13

---

Published as a conference paper at ICLR 2025
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- mto::1803113444 2018.
reinforcement learning. arXiv preprint
Tuan Nguyen, Trung Le, Nhan Dam, Quan Hung Tran, Truyen Nguyen, and Dinh Phung. Tidot: A teacher imitation learning approach for domain adaptation with optimal transport. In Inter- national Joint Conference on Artificial Intelligence 2021, pp. 2862-2868. Association for the Advancement of Artificial Intelligence (AAAI), 2021.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International Conference on Machine Learning, pp. 26228- 26224. PMLR, 2023.
Haoyi Niu, Shubham Sharma, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming HU, and Xianyuan Zhan. When to trust your simulator: Dynamics-aware hybrid offline-and-onilee reinforcement learning. In Advances in Neural Information Processing Systems, 2022. URL htt /oppeeeeeeeee...eee net/ forum? i d= 8 FOZKw.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11(--6):355-607, 2019.
Zhongjian Qiao, Jiafei Lyu, Kechen Jiao, Qi Liu, and Xiu Li. Sumo: Search- uncertainty estimation for model-based offline reinforcement learning. arXiv preprint arXiv:2408. 12970, 2024.
Roberta Raileanu, Maxwell Goldstein, and Arthur Szlam. Fast adaptation to new environments via policy-dynamics value functions. In International Conference on Machine Learning. 2020.
Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning. pp. 28701-28717. PMLR, 2023.
Dripta Raychaudhuri, Sujoy Paul, Jeroen Vanbaar, and Amit K Roy-Chowdhury. Cross-domain imitation from observations. In International Conference on Machine Learning, 2021.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems, 35:16082-16097, 2022.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, 2018.
Reda Bahi Slaoui, William R. Clements, Jakob N. Foerster, and S' S'ebastien Toth. Robust domain randomization for reinforcement learning. ArXiv, abs/1910.10537, 2019.
Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for model-based offline reinforcement learning. International Conference on Machine Learning, pp. 33177-33194. PMLR, 2023.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the min- imalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A Physics Engine for Model-based Con- trol. JEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Luca Viano, Yu ting Huang, Parameswaran Kamalaruban, and Volkan Cevher. Robust inverse inforcement learning under transition dynamics mismatch. In Neural Information Processing Systems, 2020.
14

---

Published as a conference paper at ICLR 2025
Quan Ho Vuong, Sharad Vikram, Hao Su, Sicun Gao, and Henrik I. Christensen. How to pick the domain randomization parameters for sim-to amm-to-real transfer of reinforcement learning policies? ArXiv, abs/1903.11774, 2019.
Ruhan Wang, Yu Yang, Zhishuai Liu, Dongruo Zhou, and Pan Xu. Return augmented decision transformer for off-dynamics reinforcement learning. arXiv preprint arXiv:2410.23450, 2024.
Xiaoyu Wen, Chenjia Bai, Kang Xu, Xudong Yu, Yang Zhang, Xuelong Li, and Zhen Wang. Con- trastive representation for data filtering in cross-domain offline reinforcement learning. arXiv preprint 2r0i::2405. 222444055.992.. 06/92, 2024.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy opti- mization for offiin reinforcement learning. Advances in Neural Information Processing Systems, 35:31278-33 -1221, 2022a.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning. In International Conference Machine Learning (ICML), 2021.
Theng Wu, Yichen Xie, Wenzhao Lian, Changhao Wang, Yanjiang Guo, Jianyu Chen, Stefan Schaal, and Masayoshi Tomizuka. Zero-shot policy transfer with disentangled task representation of meta-reinforcement learning. In IEEE International Conference Robotics and Automation, 2022b.
Annie Xie, Shagun Sodhami, Chelsea Finn, Joelle Pineau, and Amy Zhang. Robust policy learning over multiple uncertainty sets. In International Conference on Machine Learning, 2022.
Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Gromov-wasserstei learning for graph matching and node embedding. In International conference on machine learning, pp. 1942. PMLR, 2019.
Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross-domain policy adaptation via value-guided data filtering. In Thirty-evvntth Conference Neural Information Processing Systems, 2023. URL https: http///eeeeeeeeeeeeeeeeee net. forum? id=adM2 60dxsa.
Zhenghai Xue, Qingpeng Cai, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. State regularized policy optimization on data with dynamics shift. Advances in neural information processing systems, 36, 2024.
Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. Exploration and anti- entieexploration with distributional random network distillation. arXiv preprint arXiv:2401.09750. 2024.
Junghyuk Yeom, Yonghyeon Jo, Jungmo Kim, Sanghyeon Lee, and Seungyul Han. Exclusively penalized q-learning for offline reinforcement learning. arXiv preprint 1:XXi:2405.14085 2024.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based Offline Policy Optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative Offline Model-Based Policy Optimization. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2021.
Grace Zhang, Li Zhong, Youngwoon Lee, and Joseph Lim. Policy transfer across visual and dynamics domain gaps via iterative grounding. ArXiv, Abs/2107.00339, 2021a.
Junjie Zhang, Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, Jun Yang, Le Wan, and Xiu Li. Uncertainty- driven trajectory truncation for data augmentation in offline reinforcement learning. In ECAI 2023, pp. 3018-3025, 2023.
Qiang Zhang, Tete Xiao, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Learning cross- domain correspondence for control with dynamics cycle-consistency. In International Confer- ence on Learning Representations, 2021b. URL https: htttt:::///peeeevieee net forum?id- QTR1ze3161
15

---

Published as a conference paper at ICLR 2025
A EXTENDED RELATED WORK
Offline Reinforcement Learning (RL). In Offline RL (or batch RL) (Levine et al., 2020; Lange et al., 2012). the agent is only allowed to learn policies based on some previously gathered offline datasets. Offline RL methods can be generally divided into model-free (Wu et al., 2021; An al., 2021) and model-based (A (Argenson & Dulac-Amold, 2020; Matsushima et al., 2021) approaches, where model-free offline RL algorithms often leverage value penalization (Kumar al.. 2020; Lyu et al., 2022c; Nikulin et al., 2023; Yang et al., 2024; Yeom et al., 2024), implicit or explicit policy constraints (Fujimoto et al., 2019: Fujimoto & Gu, 2021; Kostrikov et al., 2022: Wu al., 2022a; Lyu et al., 2022a; Ran et al., 2023; Tarasov et al., 2024), while model-based offline RL methods typically rely on directly optimizing the policy with the learned model (Yu et al., 2020; Kidambi et al., 2020; Rigter et al., 2022: Guo et al., 2022: Sun et al., 2023; Qiao et al., 2024; Luo et al., 2024), or performing data augmentation for offline datasets (Yu et al., 2021; Lyu et al., 2022b; Zhang et al., 2023). The success of these offline RL methods relies heavily on the fact that the underlying static datasets usually contain a large amount of transitions. Instead, we investigate how to facilitate target domain policy training by leveraging source domain datasets, given only few target domain data.
Domain Adaptation in RL. It is challenging to generalize or transfer policies to another domain (Cobbe et al., 2019) in RL, where the two domains can differ in terms of agent embodiment (Liu et al., 2022b; Zhang et al., 2021b), observation or action spaces (Gamrian & Goldberg, 2018; Bous- malis al., 2018: Zhang et al., 2021a; Ge al., 2022), and the environmental dynamics (Eysenbach et al., 2021; Viano et al.. 2020), etc. In this work, we focus on the policy adaptation problem (Xu et al., 2023; Lyu al., 2024a:b) under dynamics shifts between the two domains, while other com- ponents like observation spaces remain unchanged. To mitigate this challenge, previous literature mainly leverages domain randomization (Slaoui et al., 2019; Mehta et al., 2019; Vuong et al., 2019; Jiang et al., 2023), system identification (Clavera et al., 2018: Du et al., 2021; Xie et al., 2022), imitation learning (Kim et al., 2019; Hejna et al., 2020; Fickinger et al., 2022; Raychaudhuri et al., 2021; Guo et al., 2024), and meta-RL methods (Nagabandi et al., 2018: 8; Raileanu et al., 2020; Arndt et al., 2019; Wu et al., 2022b), etc. Nevertheless, these methods often require expert demonstra- tions from the target domain, and prior knowledge to guide parameter randomization. Some recent works discard these demands and train dynamics-aware policies given limited offline transitions from one domain while having access to another online domain with dynamics discrepancies (Ey- senbach et al., 2021: Niu et al., 2022; Xu et al., 2023; Lyu et al., 2024a). In contrast, we explore the offline policy adaptation setting where the two domains are purely offline. Existing works address this issue via performing reward penalization (Liu et al., 2022a), training a GAN-style discriminator (Xue et al., 2024), incorporating pessimistic supported regularization (Liu et al., 2024a), utilizing the return-coneen conditiiond supervised learning (RCSL) approach (Wang et al., 2024), or conducting data filtering from the perspective of mutual information (Wen et al., 2024). Different from these meth- ods, we resort to the optimal transport approach for selecting source domain transitions that are close to the target domain, in conjunction with a policy constraint term that enforces the learned policy to stay close to the support region of the target domain. These allow our method to function well even under very few target domain samples where existing methods typically fail.
B MISSING PROOFS
In this section, we present the proofs of the theoretical results that are omitted from the main text due to space limits. We note that Q (s, a) means the state-action value function upon the sample (s, a) by following the policy π in the MDP M. Also, recall that Pare PM : Peur = PM
B. LEMMAS
Lemma B.1 (Telescoping lemma). Denote M1 = (S. A.. and M2 = (S.A.P as two MDPs that only differ in their transition dynamics. Suppose we have two policies ni, N2, can reach the following conclusion:
E B (x,0) Eg wi 127, (5',a')) - Egr Ps.a' 107, (5', a')l] JM, -m)) 1-7
(12)
16

---

Published as a conference paper at ICLR 2025
Proof. Please check the proof of Lemma C.2 in Xu al. (2023).
Lemma B.2. Denote M (S,A, P.r,P, as the underlying MDP., The performance difference of the two policies TI, in the MDP M gives:
JM( - JM(T2) = 1-y B (2.)))) Ba' [QM ( aa)] Eai ( (13)
Proof. Please check the proof of Lemma B.3 in Lyu et al. (2024a).
B.2 PROOF OF THEOREM 3.1
Theorem B.3 (Offline performance bound). Denote the empirical policy distribution in the offline
dataset Dare from source domain Mare and the offline dataset Dvar from target domain Maar as Edvr 1(s,a)
Drar (8.0)
Dar LDer 1(8)
and Dra E 1(s)
respectively. Denote C1 =
(1-a)²²
-2mmax then the return
difference of any policy between the empirical source domain Mare and the true larget domain Muar is bounded:
JM (TT Mao (T) 2 -C1 E Daro PP (TT((TD.IITT)) Ci Bpteer PM [DVV(DDual Mure
(TT ||=]]
Mure
(b): target policy deviation
(a)): source policy deviation
C1
aD Mare TDre
Prv(PMa | Mand
constant.
(c): dynamics mismatch
Proof. We establish the performance bound of the policy in the true target domain Maa and the empirical source domain Marc by dividing the performance deviation into three parts,
Maa (TT) -Mire (TT) = (JMen (T) MMar (T + (JMrar ( (T MM (TD
(0)
(6)
(14)
(Juuc (Tre - Manc (TT)
(c)
The term (a) in the RHS measures the performance deviation of the learned policy and the behavior policy of target domain dataset under the target domain, term (b) measures the performance of the behavior policy of the target domain dataset in the true target domain MDP against the performance of the behavior policy in the source domain dataset under the empirical source domain MDP, and term (c) depicts the performance deviation of the learned policy and the behavior policy in the source domain under the empirical source domain MDP. We first bound term (a). By using Lemma B.2 have
(a) JM (T)
M ( (T
Eprrear (s P (1aa) [Ea'~s ( (s',a') [[]] Ex' [0 (s',a')] a')]]
Bprze (s,a), P Pa (1a,0) Ea'NT Toer (5', (8',a')] ad)] Ea' Tow (8 a')]
Eprnen (s,a),
P P (1800)
Dru (a'ls') [aalls')) Trw (8',a')
7
a'EA
Tmax
110 1p (s,a), P Mear (1a,a)
Dau (ann(aa' (a'js')=<(()) =(a'is'))
72
2r max
7)2 Epxxe (s,0).
(TV((TDin (-18 (-16))11((1186))
17

---

Published as a conference paper at ICLR 2025
where we use the fact that [Q(s, a)
Timux 1-7"
Then, bound term (c) by using Lemma B.2,
(c) J Mice
(Dre)) Mine (T)
E Dure (s rnpp Mar (-1,,) Ea Ba'nnDero (Tinn (d',
Ba'nx (Tin (s'
7 P Mare
R. Dare (s o' Share (11,,) Mure
EaglNAD ( Mune (6', -aginar ( (8',
E Dure (s, P Paro (1s,a) Mere
(T (TD ( (1
Mar (8,
?
a'EA
T Tiax
(T (a'ls') ( =(d'(s)))
DDar (a. ro Maano (18,,))
(1 - 7)2
Pusee
a'EA
2r max
(1 7)2
Das Mare
(s.a).
Pinee
(DrV(TDend (+15 ([=1s)==
Finally, we bound term (b). By using Lemma B.1, we have
(b) Maac (TDr..)) J Manc (T
1
E DDaa (s,a) PMrl
E Ey'n a NiD
1 (5' a')a Mare
E'g' Mure a DDrr
@ (s [a,')]] Muce
?
@ (8', a') Mare
Ees alowDDaee
Tere (8', a')) Mare
7 PMaa (s,a)
(d)
+
[Bo Poor- at Dare
@ (d, Muro
Q
Bsr PMar aara
Q @ (o', (6',a'))
Moro
(e)
For term (d), it is easy find that:
(d) = E.' PM
Σ
Dra (a'ls') - x Danc (a'ls'))Q Fore Mar
(6'
LO'EA
< Extn
D (a'ls') DDr (a'1s')l 112 Menr
La'A
2r max PaI
(-1s') (-|s')]
remains bound term (e). We have
(e) Eg'n a' d' T (s', (8','') a')) Eg'n Pam wanso 1Q 12t (s',a') Mare
Eg'na
Σ (MMi (8'18,a)- -P (8 a)) DTree (8,,
( Maro
Ea'naDue
Pa (s)(s,a) PPpm (5'1s,a)] x 10 Mone (5'
Tmax En Ba
Σ (s))s,)) -
Prini (d'1s,
1
2r max Eyg Ea''niDDrr
Drv(PMan (118,) (118,))))
1 - -
18

---

Published as a conference paper at ICLR 2025
Then, get the bound for term (b):
(b) J Mene (Tar))) J Mune (mo)
max
E DDar Paaaar
(s Paar (D (-1s1]
(1 7)2
2r max
(1 (1-7)² E. Eppp (s,a) a' DTv (+|s, PM (11s,
Note that the behavior policy in the source domain and the behavior policy in the target domain are fixed, indicating that their total variation deviation under the target domain transition dynamics is constant. Combining the above bounds for term (a). term (b) and term (c), and we have
2rmax
2r 2maax E Dare Paaee Mure
Jua (T) -M (T) 2 (1 7)²
E Pi
PMeaa (TIV(T [|]]]
(1
27 max
Br
Drv P constant.
(1 7)2
MMre Dare
By replacing max with C1, we have the desired conclusion immediately. (1-
B.3 PROOF OF THEOREM 3.2
Theorem B.4. Assume that the cost is bounded, i.e., C(u, ut) S Cmax <00, V u, v. then have 0 N d(ux) 2 - -Cmax DTy (pslip))))))
Pare du, and pt. DDarl E.erol dul where Sare 2 Gsre ① & Proof. Recall that Ps [Droo Elerol
u' Star Otar star and (Sare, asre, stre) Dare: (Star, Gtar: Star) Dtar-
Note that the cost Clu, u') 2 0, V u' and the solved optimal coupling μ² is also non neganeee.te. It then easy find that
Dead
d(ue)=
Σ
O, U = (sarer (Bere) Daee.
(15)
We also have that
Davel
Darel Drad
Σ d, = W(u,
Σ Σ
(16)
t=1
1=1
1=1
Using the definition of the Wasserstein distance, we have
Derel
Darel Drar
d
min
Σ
Clue, uky)) p
NEM
DDrel Dear
min max Clue, uhr NEM
Σ
Σ
1 1lu # we ue)pee
t,t'
Dace Dearl
-max min HEM
Σ
1(2, F ur Mr))
F=
Cmax Drv (p=l(pt),
where the last equation holds due to Remark 2.27 in Peyré & Cuturi (2019). Finally, we have
DDael
dj 5
Σ de Cmaxx (rw((pll(p))) de -Cmax Drveell)
(17)
This concludes the proof.
19

---

Published as a conference paper at ICLR 2025
C ENVIRONMENT SETTING
In this section, we include a detailed description of the environmental settings we adopt in this paper, such as the basic information on the source domain datasets and target domain datasets, details on how dynamics shifts are realized, etc.
C.I DATASETS
Source domain datasets. Since we consider the setting where only few target domain data are available, we directly adopt the MuJoCo datasets from D4RL (Fu et al., 2020) as source domain datasets. In D4RL, the MuJoCo datasets are collected during the interactions with the continu- ous action environments in Gym (Brockman et al., 2016) simulated by MuJoCo (Todorov et al., 2012). We adopt four tasks, halfcheetah, hopper, walker2d, ant, and consider source domain dataset qualities across medium, medium-replay, medium-experr. The medium datasets contain experiences collected from an early-stopped SAC policy for 1M steps. The medium-replay datasets record the replay buffer of policy trained up to the performance of the medium agent. The medium-exper datasets are constructed by mixing the medium data and expert data at 50-50 ratio. Note that the source domain datasets can have quite distinct dataset sizes, e.g., medium datasets have 1M samples while medium-replay datasets can only have 100K samples.
Target domain datasets. To examine the offline policy adaptation capability of our method, we design three kinds of dynamics shift scenarios for empirical evaluations based on four widely used MuJoCo tasks (HalfCheetah-v3, Hopper-v3, Walker2d-v3, Ant-33), including gravity shift, kine- matic shift and morphology shift. The gravity shift means that the gravitational forces acting on the robot in the source and target domains are different, the kinematic shift indicates that some joints of the simulated robot are broken in the target domain, while the morphology shift suggests that there some morphological mismatch between the simulated robot in two domains. We show the visu- alization results of the agent in the source domain and the target domain in Figure 4. We explicate the detailed modifications in the following subsections.
We then collect target domain datasets in the revised environments by following a similar data- collecting procedure as D4RL. We train an SAC (Haarnoja al., 2018) agents environments with dynamics shifts for IM steps. We log the policy checkpoints of the agent during training and use them for rolling out trajectories. The expert datasets are generated using the last policy checkpoint, and the medium datasets are gathered with the policy checkpoint that exhibits approximately 1/2 or 1/3 the performance of the expert policy. To ensure that only limited budget of target domain data can be accessed, we only collect 5 trajectories for each dataset, which amounts to about 5000 transitions. We do not follow D4RL, to simply merge the medium dataset and the expert dataset to produce medium-exper datasets. Instead, we strictly follow the data budget and pick trajectories from the medium dataset and trajectories from the expert dataset to construct the medium expert datasets. We observe that it is extremely difficult for off-the-shelf offline RL methods acquire meaningful performance given such little data. Since we evaluate the performance of the agent in our modified environments, to better characterize the performance of the agent across different tasks. The normalized score performance of the agent in the target domain gives:
follow D4RL and propose use the normalized score metric
J Jr 100, NS
(18)
Je Jr
where J is the return acquired by the agent in the target domain, Jr, Je are the returns obtained by the random policy and the expert policy in the target domain, respectively. We summarize the reference scores of J, and Je under different dynamics shift scenarios in Table 3. We also list the minimum return, the maximum return, and the average return of trajectories each target domain offline dataset in Table 4.
C.2 GRAVITY SHIFT TASKS
To simulate the gravity shifts between the source domain and the target domain, modify the xml files of the underlying environments. We set the gravitational acceleration of the target domain to half of that in the source domain, and we do not alter the direction of the gravitational force.
20

---

Published as a conference paper at ICLR 2025
Source domains
Target domains (gravity shifts)
Target domains (kinematic shifts)
Target domains (morphology shifts)
Figure 4: Illustration of the adopted environments. Target domain robots have gravity shifts (second row), kinematic shifts (third row), and morphology shifts (bottom) compared with the source domain (top) robots.
Table The referenced min score and max score for MuJoCo datasets under various dynamies shift scenarios. These are used to evaluate the normalized score performance of the algorithm in the target domain. Since the source domain and the target domain differ from each other in transition dynamics, one could not simply adopt the reference scores from D4RL directly.
Task Name
Dynamics shift type
Reference min score J, Reference max score Je
halfcheetah halfcheetah
gravity
280.18
9509.15
kinematic
280.18
7065.03
halfcheetah
morphology
-880.18
9713.59
hopper
gravity
-26.336
3234.3
hopper
kinematic
26.336
2842.73
hopper
morphology
26.3336
3152.75
walker2d
gravity
10.08
5194.713
walker2d
kinematic
10.08
3257.51
walker2d
morphology
10.08
4398.43
ant
gravity
3325.6
4317.065
ant
kinematic
-325.6
5122.57
ant
morphology
-325.6
5722.01
halfcheetah hopper walker2d anl.ger/d//tttggviitt: the modifications of the xm1 file gives:
gravity
coption gravity-"O -4.905" t mimestep="0.0111
C.3 KINEMATIC SHIFT TASKS
Different from the gravity shift, the kinematic shift occurs at different parts of the simulated robot. Detailed modifications of the xml files give:
halfcheetah-kinemadl The rotation angle of the joint on the thigh of the robot's back leg is modi- fied from [00.22, 1.05) to [-0.0052, 0.0105).
21

---

Published as a conference paper at ICLR 2025
Table Trajectory return information of the target domain datasets. We list here the minimum return (abbreviated as min return), the maximum return (denoted as max return), and the average return of trajectories in the target domain datasets.
Task Name
Dynamics shift
Dataset type
Min return
Max return
Average return
halfcheetah
gravity
medium medium-expert
4179.82
4383.32
4296.27
halfcheetah
gravity
4342.78
8243.03
6567.94 .94
halfcheetah
gravity
expert
7846.18
8339.18
8131.54
halfcheetah
kinematic
medium
2709.52
2782.61
2755.50
halfcheetah
kinematic
medium-expert
2709.52
7065.04
5298.61
halfcheetah
kinematic
expert
6951.27
7065.04
6998.93
halfcheetah
morphology
medium
4070.43
4214.64
4156.88
halfcheetah
morphology
medium-expert
4070.43
9713.60
7475.13
halfcheetah
morphology
expert
9474.34
9719.81
9614.92
hopper
gravity
medium
1784.88
2885.13
2367.66
hopper
gravity
medium-exper
2416.82
4143.63
3297.79
hopper
gravity
expert
3745.59
4186.19
4051.07
hopper
kinematic
medium
1849.06
1886.89
1870.16
hopper
kinematic
medium-expert
1868.20
2842.17
2452.67
hopper
kinematic
expert
2840.97
2842.73
2841.83
hopper
morphology
medium
1946.73
2039.80
1980.33
hopper
morphology
medium-expert
1980.06
3152.75
2694.58
hopper
morphology
expert
3148.26
3152.75
3151.39
walker2d
gravity
medium
2421.98
3444.63
2897.85
walker2d
gravity
medium-expert
3144.32
5166.62
4415.11
walker2d
gravity
expert
5159.51
5219.14
5174.51
walker2d
kinematic
medium
1415.69
2223.17
2026.49
walker2d
kinematic
medium-exper
1415.69
3257.51
2442.82
walker2d
kinematic
expert
2874.92
3257.51
3077.19
walker2d
morphology
medium
772.82
2875.40
2013.11
walker2d
morphology
medium-expert
772.82
4348.94
2961.67
walker2d
morphology
expert
4341.38
4398.44
4354.58
ant ant gravity
gravity
medium
377.10
3247.66
2314.45
medium-expert
377.10
4511.55
2131.79
ant
gravity
expert
335.28
4584.53
3365.35
ant
kinematic
medium
2826.00
3111.93
3017.82
ant ant kinematic
kinematic
medium-expert
2826.00
5122.58
4240.99
expert
5009.82
5122.57
5072.50
ant morphology
medium
2150.50
2204.32
2176.02
ant morphology
medium-expert
5150.50
5653.05
4199.54
ant morphology
expert 5461.49
5722.01
5586.73
broken back thigh
jjoint axis="O 1 0" damping="6" name="bthigh" pos="0 0" range="". .0522 .1155" sti stiffss==="2A0 type="hinge"/
hopper-kinematic: The rotation angle of the head joint is modified from |-50 to [-0.15, and the rotation angle of the foot joint is modified from [-45, 45] to [-18, 18).
hoad joint
<joint axis="0 -1 0" htme="thigh_jee pos="O 0 1.05" range=" 00.1 type="hinge"/>
foot joint
<joint axis="0 -1 0" name= foot_ pos="0 0.1" range="-18 18" type= "hinge"/>
22

---

Published as a conference paper at ICLR 2025
walker-kinematic: We modify the rotation angle of the foot joint the robot's right leg from |-45, 45] to 0.455.
right foot <joint axis-"O -1 type="hinge"/>
" foot_jointt pos-"O 0. range=" -0.45 0.45"
ant-kinematic: The rotation angles of the joints on the hip of two legs in the ant robot are modified from [-30, 30] to [-0.3,0.3
hip joints of front legs
<joint axis-"O 00 1" name- "hip_"" 1_ pos- "0.0 0.0 0.0" range="-O0 -0.3 0.3" type"
"hinge"/> />>
<joint axis="O 1" name="hip 2" pos="0.0 0.0 0.0" range="-O0 -0.3 0.3" type= "hinge"/>
C.4 MORPHOLOGY SHIFT TASKS
Akin to the kinematic shift tasks, the morphological change of the robot in the target domain differs per environment. To be specific,
malfcheetah-morph:: We modify the sizes of the back thigh and the forward thigh the Cheetah robot below:
back thigh
<geom fromto-"0 0 0 -0.0001 0 -0. 0000011" name-"bthigh. size="0.048 type-"
capsule"/>
<body name="bshin" pos=*-0.0001
-0.0001">
forward thigh
<geom fromtc="O 0 0.0001 0 o. 0001"" name="fthigh" size="0.045 type="
capsule"/>
<body name=" pos=*0. 0001 0 0.0001">
hopper-morph: We increase the head size of the robot. The modifications shown below:
head size
<geom fricticn="O.9 9" fromto="0 0 1.45 0 1.05"
name="torso_geom size="
0.125" type="capsule"/>>
walker-morph: We modify the thigh the right leg of the robot the following:
right leg
<body name="thigh" pos="0 1.05">
<joint axis=*0 -1 0" name="thigh joint" pos="0 1.05" range="-150 0"
type="hinge"/>
<geom fricatin="0.9" fremto="O 1.05 0 1.045" name="thi name="thigh__eemm geomm size- "0.05" type="capsule"/>
<bedy name="leg" pes="0 0 0.35">
<joint axis="0 0" name="leg joinn pos="0 1.045" range="-150 0"
joint"
type="hinge"/>
<geom friction="O.9" fromto="0 1.045 0 0 0.3" name-"leg_gee size-"
0.04" type="capsuie"/
<body name="foot" pos=*0.2 0">
<joint axis="0 -I 0" name=" fame="foot_joint" pos="O 0 0.3" range="-45 45"
type="hinge"/>
<geom friction="O.9 fromto="-0.0 0 0.3 0.2 0 0.3" name= fame="foot_goe foot__geo"'
size="0.06 type="capsule"/>>
</boay>
</body>
</body>
ant-morph: We reduce the size of the ant robot's feet on its front two legs to simulate an ant robot with short feet. The detailed modifications
23

---

Published as a conference paper at ICLR 2025
front Leg 1
<geom fromto= .0 0.0 0.0
0.1 0.1 0.0" name=" Leme==Lett__nnle
ankle geom" size="0.08
type="capsule" carccpppule>>>>
front leg 2 <geom fromte-"O.0 0.0 0.0 -0.1 0.1 0.0" name same-"right_akkiee size="
0.03" type=" capsulee>>
We have included the modified xml files in the supplementary material.
D IMPLEMENTATION DETAILS
In this part, we describe the implementation details of the baseline methods and our proposed OTDF algorithm. We also provide detailed pseudocode for OTDF+IQL. Furthermore, we list the hyper- parameter setup used for methods.
D.1 BASELINES
IQL*: IQL (Kostrikov et al., 2022) is a widely used offline RL algorithm that learns an in-sample policy without querying OOD samples that lie outside of the offline datasets. Since observe that IQL can not learn meaningful policies merely on the target domain dataset, we slightly alter its training objectives by involving both the source domain dataset and the target domain dataset. We name the revised method IQL* It trains the state value function via expectile regression:
Lv ((s.a)) (0
(O - (s)())),
(19)
where - -r- 1( < 0) w2 I(.) is the indicator function, 0' is the target network parameter. The state-action value function is then updated by:
Lq E(s.arrs) [(r(s, a) (V) Qo(s,
(20)
We then calculate the advantage function A(s, = Q(8 a) V(s) and use it as weights for policy optimization:
Lactor B(s.))Daa Un (exp(S1qu. x A(s, a)) log {p(a))))
(21)
where FIQL is the inverse temperature coefficient. We implement IQL by following its official codebase". We adopt the symmetrie sampling method when sampling data from two offline datasets.
DARA: DARA (Liu et al., 2022a) is the offline version of DARC (Eysenbach et al., 2021). trains two domain classifiers 4s (target) St. Oe, S1+1), 90 (target) st, at) with the following objectives
L(OSAA) p... [log gosss (target) (8t, (t, 81+1)] ED... [log(1 90sAs (target 81, at, 84+1)) 1))] C(AsA) = E Drar [log A (target) (s,, at)] + Ep [log(1 9asa (target) s4, at))],
PM uls).e))
to estimate the dynamics gap log PM (#c+1 a.. between the source domain and the target domain. DARA approximates this term by leveraging the trained classifiers and proposes modify the source domain rewards as follows
4esAs (target st, ae: 8t++ )966 (source at) (22)
ÍDARA = x dr, Ör(s;, a.) log
4asas (source) St, ass Se+ 49 (Larget/ss, ax)"
where À is an important hyperparameter that controls the strengths of the reward penalty, We em- pirically find that setting À or larger value leads to often incurs quite poor performance. We hence set = 0.1 for DARA by default. We implement DARA by following http:///ppenreiie.neefffoou 2id-9SDQB3b6SK We clip the reward penalty term to lie in [-10, 10] for training stability. We use IQL as the base algorithm for DARA to align with other algorithms.
code attached in
BOSA: BOSA (Liu et al., 2024a) proposes to address the cross-domain offline RI problems by including two support constraints, which handle the OOD state actions problem through a supported
hhtgs///gthhh.ccm//oorrrw////iip q learning
24

---

Published as a conference paper at ICLR 2025
policy optimization and mitigate the OOD dynamics issue through supported value optimization. To be specific, the critics in BOSA are updated via:
Leritic (s.a)~Dare
(s,
E (8,0,r,s) Dun a' d'mxx(+11)
[PPaar(S a)) > (Qa a) - v)²
-1)
(23)
where 1(-) is the indicator function, Piar (aa\ss/ss,, ss,a) arg max ((s,"" [log Pt (s' a)] is the estimated target domain transition dynamics, is the selection threshold, i E {1, The policy in BOSA is updated via another supported optimization objective:
Lactor =
eexra)) (s, a)), s.t. De.-D.. Memm (TT > è'
(24)
where e' is a hyperparameter that determines the selection threshold, femis is the learned behavior policy of the mixed dataset Dere U Dear- BOSA models the transition dynamics model in the target domain and the behavior policy of the mixed dataset with CVAE. Since open-source codes can be found for BOSA, we implement it by following the instructions in the original paper. BOSA is trained for 1M gradient steps in practice by drawing samples from both the source domain dataset and the target domain dataset. We use SPOT (Wu et al., 2022a) as the backbone for BOSA.
SRPO: SRPO (Xue al., 2024) proposes optimize the policy by solving the following con- strained optimization problem:
max Ex Σ 7'r(01, at) s.1. (dI((e)))I)( <e,
(25)
IT
where Tx is the trajectory induced by the policy n, (-(-) the stationary state distribution of policy π. (()) denotes the optimal state distribution in other environment dynamics. The above problem can transformed into the unconstrained optimization problem via Lagrange multipliers, where the logarithm of probability density ratio A log
((x)) is added to the vanilla reward term. In light d=().)
of this, SRPO samples batch of data with size N from two offline datasets Dere. Dear and ranks these transitions by state values. SRPO then tags proportion of pN samples with high state-values as real data and others as fake data. It trains discriminator Da(-) to distinguish these samples, and proposes to modify the rewards via:
ÁSRPO + Ds(8)
(26)
Ds)s)
where A is the reward coefficient. We use a fixed proportion of real data 0.5 for all experiments. We do not the find official code for SRPO and implement it ourselves by following its original paper.
IGDF: IGDF (Wen et al., 2024) captures the dynamics gap between the source domain and the target domain through contrastive learning. It trains score function h(:) using (s, a, s(ac) D from the target domain as positive samples, and mixed transition (s, a, sare)) as negative samples, where (s. Dtar- Sare objective:
Derc The score function is optimized via the following contrastive learning
h(s, a, S(
Ccontrastive = -E(s, Est-
log Σe a(s,a,s')
(27)
where S"- is collection of the next states in negative samples. Practically, IGDF adopts two neural networks (fs, a), (ss)) to learn representations of state-action pairs and states, and approximate the score function with linear parameterization of them:
h(s, s') exp(o(s, (s))))
(28)
Based on the measured score function, IGDF proposes to filter out source domain data when training value functions:
1 1 D [(e (Q0- (Q0)) a-ss,, (E (h((s,a,s)) > he% (QQ - T
Laritic 2
(29)
where is the importance coefficient for weighting the TD error of the source domain data, & is the data selection ratio akin to that in OTDF. We run IGDF by using its official codebase" and use IQL as its backbone.
hhtgs://gittuh.co/BBtteaeaaaa Wenn//GD
25

---

Published as a conference paper at ICLR 2025
D.2 ALGORITHMIC DETAILS OF OTDF
To avoid solving repetitive OT problems across different seeds of the same task, we solve the OT problem before the policy training process begins. This is feasible thanks to the fact that both the source domain and the target domain are offline. We use the cosine distance as the cost function in OTT-JAX. After calculating the optimal coupling, we measure the deviation of the source domain data to the entire target domain dataset via Equation 5, This process is very computationally efficient thanks to the OTT-JAX library, solving the entropy-regularize OT problem within minutes given 1M source domain data and 5000 target domain data. Then we incorporate these deviations into the source domain dataset. For practical usage, we build the practical OTDF algorithm on top of IQL.
We summarize the pseudocode of OTDF+IQL in Algorithm 2. When updating the value function, we selectively reject some source domain data that deviate far from the span of the target domain dataset. We further normalize the deviation obtained by OT with min-max normalization and lever- age them as weights for source domain data. This encourages the agent to adaptively emphasize source domain data that lie close the target domain while de-emphasizing others. This can also be viewed as a regularization term that penalizes the source domain transitions. Note that the data filtering process only occurs the optimization process of the state-action value function Q and the update formula of the state-value functfnn V remains unchanged.
Algorithm Optimal Transport Data Filtering (OTDF)
Input: Source domain dataset Dare. target domain dataset Dear, batch size N, data selection ratio 1: Initialize policy network TΦ. value networks Vy. Qo. target Q function Qor, the cost function
C, CVAE G. policy coefficients 3, number of sampled latent variables M, target update rate r)
2: // Solve the OT problem
3: Compute the optimal alignment between Dare and Diar with Equation 4
4: Compute deviations {de ) between the source domain data and Drar with Equation 5
5: Concatenate Darc and [de)
Purel to get Darc {( (ts Tty & de) ]
DDerl |Prel
6: // CVAE training
7: Train CVAE policy to model the behavior policy in the target domain dataset with Equation 10
8: for i 1, 2, do
9: Sample a mini-batch bare == {(s, r, s', s',)) with size N from Dare
2 10: Sample mini-batch bea == [(ss[[ T. s')} with size N
2 from D
Update the state value function V/ via: Lv = Da..a)DDe (O (s, (s)(
12:
Normalize the deviations d via Equation 8 obtain normalized deviations d // Data filtering
13:
14:
Rank the deviations of the sampled source domain data and admit top E% of them
Compute the weights for the remaining source domain data via exp(d)
16:
Compute the target value via: y = r + (s')
17:
Optimize the state-action value function Qo on byre U btar via:
Lq Epar [(Qo y)² + E (e.anrs ()) (exp()))(d> (0 y)2]
18:
Update the target network via: 0' - 110 + (1-1)0' // Dataset regularization
20:
Decode M actions from CVAE and construct Gaussian distributions @ (+1s) Compute the advantage A and optimize the policy Ty. on bare Ubtr using advantage-weighted regression (AWR) and dataset regularization:
M m (T
La
E Den
[exp(BiQr 7a)l)=)((
E DerereDDe
log
22: end for
As for the policy update, we introduce novel dataset regularization term in conjunction with the vanilla weighted behavior cloning term in IQL. We adopt the conditional variational autoencoder (CVAE) approximate the behavior policy in the target domain dataset. Note that find that
26

---

Published as a conference paper at ICLR 2025
CVAE can well-model the target domain behavior policy even given limited target domain data. This phenomenon can also be observed in the single-domain model-free RL, e.g., BCQ (Fujimoto et al., 2019) can achieve quite good performance contains an encoder F and a decoder D. Both the encoder and the decoder contain two intermediate layers with 750 hidden units in each layer. We use the relu activation for each intermediate layer.
Adroit human datasets from D4RL. The CVAE G
We note that the dataset regularization term Equation 11 can degenerate into:
M
L = f.t -
IE
log
exp(log क5 ( (=(((())))))
om Dare Dear
Lxx 3x
log
ta t (+({s))ss)
5mw Dese
D.3 HYPERPARAMETER SETUP
We summarize the detailed hyperparameter setup for all baseline methods and OTDF in Table 5. For SRPO, we report its best performance by sweeping its reward coefficient À across {0.1, 0.3). For IGDF, we set its data selection ratio 5% 75% we find setting it to be 25% incurs poor performance. We sweep the representation dimension in IGDF across {16, 64) and report the best score, As for OTDF, we adopt fixed 5% 80% for all tasks. We set the policy coefficient 3 = 0.5 for all of our experiments except for all halfcheetah and walker2d tasks under gravity shifts, where we use 3 = 0.1. We do not bother tuning the hyperparameters and demonstrate that our method can achieve quite strong performance with one set of hyperparameters under many scenarios.
E WIDER EXPERIMENTAL RESULTS
In this section, we provide wider experimental results that omitted from the main text due to the page limit. We present the comprehensive normalized score comparison of OTDF against other baselines under tasks with kinematic shifts. We also investigate whether the advantages of OTDF can still hold when expert-level source domain datasets are provided. Furthermore, we study whether it necessary to adaptively weight source domain data.
MISSING RESULTS UNDER KINEMATIC SHIFTS
We summarize the normalized score comparison of OTDF against other baselines under the kine- matic shift tasks Table 6. As shown, OTDF achieves the best performance on 22 out of 36 tasks while remaining competitive against other methods on the rest of the tasks. OTDF achieves a to- tal normalized score of 1547.6, surpassing IQL* by 29.7% % and the second best approach (IGDF) by 21.8%. Again, we observe that existing cross-domain offline RL methods often fail to bring performance improvement compared to IQL*, and OTDF is the only algorithm that significantly outperforms them. We believe these further verify the superior offline policy adaptation capability of our proposed OTDF algorithm.
E.2 CAN OTDF BEAT BASELINES GIVEN HIGH-QUALITY SOURCE DOMAIN DATASETS?
In the main text, we only consider source domain datasets with medium, mediun-replay and medium- expert qualities. These datasets may typically have broader coverage and possibly contain many transitions that are similar to those in the target domain datasets. However, it can also happen in real-world applications that we may get access to sufficient expert source domain datasets. The expert source domain datasets often have narrow distribution and state-action coverage. It is then interesting to examine whether OTDF can still beat baseline methods under such a scenario.
We then adopt the D4RL MuJoCo "-v2" expert-level datasets as source domain datasets. We allow the quality of the target domain dataset to be medium, medium-expert, or expert. We choose the morphology shift tasks and all algorithms upon them for 1M gradient steps across 5 random seeds. The experimental results are shown Table 7. can be seen that OTDF outperforms other methods on 6 out of 12 tasks. OTDF achieves total normalized score of 393.0, while the second
27

---

Published as a conference paper at ICLR 2025
Table 5: Hyperparameter setup for OTDF and baseline methods.
Hyperparameter
Value
Shared
Actor network
(256, 256)
Critic network
(256, 256)
Learning rate
10
Optimizer
Adam (Kingma & Ba, 2015)
Discount factor
0.99
Nonlinearity
ReLU
Target update rate
x 10
Source domain Batch size
128
Target domain Batch size
128
IQL
Temperature coefficient Maximum log std Minimum log std Inverse temperature parameter SIQL
0.2
2
20
3.0
Expectile parameter
0.7
DARA
Temperature coefficient
0.2
Classifier network
(256, 256)
Reward penalty coefficient
0.1
BOSA
Temperature coefficient
0.2
Maximum log std
2
Minimum log std
20
Policy regularization coefficient Apolicy
0.1
Transition coefficient Avansition
0.1
Threshold parameter e,
(0g(0.01)
Value wight
0.1
CVAE ensemble size of the dynamics model
5
SRPO
Discriminator network
(256, 256)
Data selection ratio Reward coefficient
0.5
0.31,
IGDF
Representation dimension
{16, 64) (256, 256)
Contrastive encoder network
Encoder pretraining steps
7000
Importance cocfficient
1.0
Data selection ratio 5%
75%
OTDF
CVAE training steps
10000
CVAE learning rate
0.001
Number of sampled latent variables M
10
Standard deviation of Gaussian distribution
VO.1
Cost function
cosine
Data filtering ratio 5%
80%
Policy coefficient 8
{0.1, 0.5)
28

---

Published as a conference paper at ICLR 2025
Table Performance comparison under the kinematic shift. half halfcheetah, hopp = hopper, walk walker2d, m medium, r= replay, expert. The target column denotes the offline dataset quality of the target domain. We report normalized scores and standard deviations in the targer domain under varied dataset qualities of the source domain data (medium, medium-replay, medium- expert) and target domain data (medium, medium-expert, expert). The results are averaged over seeds. We bold and highlight the best cell.
Source
Target
IQL
DARA
BOSA
SRPO
IGDF
OTDF (ours)
half-m
medium
12.3
10.611.2
8.3
16.8+4.2
23.6=5.7
40.2=0.0
half-m
medium-experr expert
10.811.
12.912.8
8.7
10.312.7
9.8+2.4
10.1=4.0
half-m
expert
12.61:1.7
12.1#1.0
1084:17
12.240.9
12.8-0.7
8.742.0
half-m-r
medium
10.0+:5.4
11.5++4.9
7.5+3.1
10.2+:3.7
11.6=4.6
37.8+2.1
half-m-r
medium-expert
6.5+3.1
9.2+44.7
6.6+1.77
9.5+1.8
8.6+2.33
9.7-22.0
half-m-r
expert
13.6+11.4
14.8+22.0
10.4+4.9
14.8+-2.2
13.9-2.2
7.2+1..4
half-m-c
medium
21.8++6.5
25.947.4
30.0+:4.3
17.2+:..
21.9=6.5
30.7=9.6
half-m-e
medium-expert
7.6.11.4
9.514.22
6.8.2.9
9.6.1..4
8.913.3
10,9
half-m-e
expert
9.112..4
10.41.3
4.913.2
11.2.11.0
10.7-1.4
3,2.1.66
hopp-m
medium
58.7.18.4
43.9115.2
12.3.66.6
6..4
1..
65.6.11.9
hopp-m
medium-cxpert
68.5+12.4
16.4+16.9
15.6
43.9+30.8
51.1=18.5
55.4=2..1
hopp-m
expert
79,9:33555
83.7
14.845.5
33.1
87.4=25.4
19.0=1994
hopp-m-r
medium
36.0+t.11
39,47..2
3.2+2.6
36.1=0.2
35,9=2,4
12.5
hopp-m-r
medium-expert
36.1+.0.1
34.143.6
4.4-2.88
36.0+0.1
36.1=0.1
47.5
hopp-m-r
expert
36.0++0.1
36.1+0.2
3.7+2.5
36.1+0.1
36.1=0.3
39.5
hopp-m-e
medium
66.0+-0.5
61.144.0
35.0+-20.1
64.6+-2.6
65.2-1.5
65.3+22.4
hopp-m-e
medium-expert
45.1
61.9116.9
13.91499
14.7
15.9-15.6
38.6
hopp-m-e
expert
44.9119.8
84.212111
12.01:.33
47.61100.
$2.8-39.7
19.9
walk-m
medium
34.3:19.8
35.2122.5
14.3111.2
39.016.7
41.9-11.2
49.6-18.00
walk-m
medium-expert
30.2+12.5
51.9+11.5
13.6+7.7
38.6+6.5
42.3-19.3
43.5+16.4
walk-m
expert
56.4+18.2
40.7+14.4
15.3+2.
17.3+12.2
60.4=17.5
46.7-=3.6
walk-m-r
medium
11.5+7.1
12.5+4.3
1.9-2.1
14.3+3.1
22.2-5.2
49.7=9.77
walk-m-r
medium-expert
9.713.88
11.21..0
4.6.3..0
4.2.1..1
7.6.1.99
55.9
walk-m-r
expert
7.7144.8
7.412.4
3.6.1.55
13.2.8.5
7.512.11
51.9
walk-m-e
medium
41.8118.8
38.1114.4
21.418.3
36.914.3
11.2--3.0
44.6
m-e
medium-exper exppert
22.2428.7
23.648.1
15.914.1
23.217.9
28.1=4.0
16.5#7.2
walk-m-e
expert
26.3:1104
36.019.2
18.5+3.6
40.919.6
46.2-19.4
42,4=9.1
ant-m
medium
50.0:15.6
42.347.6
20.9+2.6
50.5
14.5=113
55.4=0.0
medium-expert
57.8+7.22
54.1+3.8
31.7+7.0
54.9+11.
54.5-4.6
60.7+3.6
ant-m
expert
59.6+-18.5
54.2+1113
45.4+.866
45.5+-9.3
19.4-14.6
90.4
ant-m-r medium
43.7+-4.6
42.045.4
19.0+1.8
45.3+-5.1
41.455.0
52.8
ant-m-r
medium-expert
36.5115.9
36.016.77
19.1
36.216.6
37.2-4.7
54.2.1.2
ant-m-r
expert
24.4.14.8
22.1110.4
19.510.8
27.113.77
24.3-2.8
10.5
ant-mn-e
medium
49.5114.1
44.7144.3
19.018.0
41.318.1
41.8-8.8
50.2.14.3
ant-m-c
medium-expert
37.24:2.0
33,3+7.0
6.4-2.5
32.8+8.0
41.5=4.9
48.8=2.7
ant-m-e
expert
18.748.1
17.8+23.6
14.5+9.0
35.2+15.5
14.4-22.9
78.4
Total Score
1193.0
1219.8
513.5
1195.7
1271.0
1547.6
best approach, DARA, only achieves total normalized score of 298.1. Despite that OTDF exhibits inferior performance on some tasks here, we would argue that all hyperparameters adopted in OTDF fixed, i.e., 5% 80%, 3 0.5, without any hyperparameter tuning. Nevertheless, tune hyperparameters for baseline methods. We strongly believe that the performance OTDF can be further improved by carefully tuning the data selection ratio and the policy coefficient.
F.3 IS NECESSARY TO WRIGHT SOURCE DOMAIN DATA?
In this part, we provide an ablation study to investigate whether we should adaptively weight the source domain data with exp(d). To be specific, we consider variant of OTDF that leverages the following objective function to update its state-action value function:
Lo ED.... [[Q> T (
B(a.,,")
[1dd dex TQ
(30)
29

---

Published as a conference paper at ICLR 2025
Table 7: Performance comparison under the morphology shift tasks given expert-level source domain datasets. half = halfcheetah, hopp hopper, walk walker2d, e expert. All methods are run over varied random seeds. We report normalized scores along with the corresponding standard deviations in the targer domain given different qualities of target domain data (medium, medium-expert, expert). We bold and highlight the best cell.
Source
Target
IQL
DARA
BOSA
SRPO
IGDF
OTDF (ours)
half-e medium 1..1=1.0
10,9
40.5
40.8:1.0
39,41:266
3..6:1.4
half-e
medium-expert
22.5=3.6
27.9
28.7+2.7
23.113.9
24.31:2
27.413.6
half-e
expert
7.9+1.5
8.6+00.5
8.0+0.7
6.3+0.8
7,411.3
8,8+
hopp-e
medium
9.5+22.3
11.4+0.55
8.7+1.77
10.5+1.1
9.6+3.66
5.7+1.2
hopp-e medium-cxpert
9.8+22.8
10.0
8.5+1.8
10.7
11.5++0.4
6.5
hopp-e expert
10.3:1..9
9,6.1399
8,4.3..3
11.9.10.3
10.2.12.8
5.7.14.0
walk-e medium
36.7.1..6
36.914.5
6.1.5.11
36.717.7
38.61110.2
32.4115.1
walk-e
medium-expert
20.6.1.2
29.2.19.0
4.5.1..9
21.915.6
30.1155.9
34.8.18.8
walk-e
expert
16.4=10.9
30.0=15.7
11.4
21.948.1
32.8+22.4
42.5+17.1
ant-e medium
31.0=9.0
39.6#2.5
33.0
33.5
35.8+8.1
1,
ant-e medium-entt
28.1=4.2
37.8+9.55
47.3+13.8
17,4
36,1#3,8
6,.4+688
ant-e
expert
4.0.-12.0
16,2+11.8
69.347.8
18.7446.1
4.0+77.5
68.8
Total Score
236.9
298.1
274.4
273.4
279.8
393.0
w/o weight
vanilla
hafchholaa-Knnemmt
hoppar-Ainomatic
walkor2d-mmarph
antimmorph
Retum
108
3000
00
04
04
04
08
10
80
0.4
0.
0.
044 04
10
Staps
Million Sleps
Millon Slaps
Staps
Figure 5: Ablation study on the source domain data weight. *-kinematic denotes tasks with kinematic shifts and *-morph means environments with morphology shifts. w/o weight refers to OTDF that excludes the component of source domain data weight, and vanilla denotes the vanilla OTDF algorithm. We report average returns in the target domain across 5 different random seeds and the shaded area captures the standard deviation.
Compared to Equation 9, the above objective function treats each filtered data with equal weights. Intuitively, this can be problematic since the data selection ratio 5% constant, and bad transi- tions can still be included for training if no adaptive weighting mechanism (or regularization on the source domain data) involved. Empirically, we conduct experiments on some selected tasks (two kinematic tasks and two morphology tasks) using the medium-replay source domain dataset and the expert target domain dataset. We present the experimental results in Figure 5. evident that the vanilla OTDF beats OTDF w/o weight on out of tasks, indicating that incorporating the weights exp(d) is a better choice.
F COMPUTE INFRASTRUCTURE
In Table 8, we list the compute infrastructure that to run all of the algorithms.
Table 8: Compute infrastructure.
CPU
GPU
Memory
AMD EPYC 7452
RTX3090
288GB
30


Reviews:
[{"review_id": "46Xo7xB8aw", "reviewer": "Reviewer_i1Jo", "summary": "This paper proposes a novel method named Optimal Transport Data Filtering (OTDF) for cross-domain offline policy adaptation. Previous works still need a comparatively large target domain dataset to learn domain classifiers or filtering via mutual information. In this paper, the authors provide a performance bound to identify the factors of the performance deviation. This paper proposes to use optimal transport to align data distributions and a regularization term to ensure the learned policy remains within the target domain’s span. The empirical study shows that OTDF outperforms existing methods in cross-domain offline policy adaptation.", "questions": "- Can you provide more insights on using the small amount of target domain data in the proposed method? Can you provide a more detailed explanation from the theoretical and empirical sides?\n- Can you explain more about the CVAE you used in the proposed method? Why use the CVAE to estimate the behavior policy? How does the CVAE affect the performance of the proposed method?\n- Can you discuss and cite more related works on cross-domain policy adaptation such as [1], [2], and [3], and how the proposed method compares to these methods?\n- Can you explain why use the different performance level target domain dataset from the source domain dataset in the empirical study? For example, why use the medium dataset in the source domain combined with the expert dataset in the target domain? Is it a reasonable setting for cross-domain policy adaptation?\n- Do you have any results for other offline RL backbones like CQL, BCQ, or BEAR? How does the proposed method compare to these methods in cross-domain offline policy adaptation?\n\n[1] When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning\n\n[2] Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation\n\n[3] Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a theoretical analysis of the performance bound in cross-domain offline policy adaptation, providing a clear foundation for the proposed method.\n- The optimal transport-based data filtering method is motivated and efficient to compute the Wasserstein distance between source and target domains and filter the source data accordingly.\n- The empirical results across various environments and types of dynamics shifts demonstrate that OTDF significantly outperforms baseline methods based on the IQL.", "weaknesses": "- The proposed practical implementation of OTDF still needs to learn a conditional variational auto-encoder to estimate the behavior polic which may introduce additional computational overhead.\n- The paper argues that the proposed method can use a small target domain dataset however the detailed explanation of theoretical and empirical analysis is still limited.\n- Ablation study on the proposed method is limited and the comparison with other dynamic model-based methods or adaptive transfer learning approaches is not comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CA1IVM9DE1", "reviewer": "Reviewer_6ito", "summary": "This paper addresses the challenge of cross-domain offline reinforcement learning (RL) where the goal is to leverage a source domain dataset to improve policy learning in a target domain with limited data. This paper identifies that directly merging data from different domains can lead to performance degradation due to dynamics mismatches. To mitigate this, they propose a method called Optimal Transport Data Filtering (OTDF), which aligns transitions from source and target domains using optimal transport, selectively shares source domain samples, and introduces a dataset regularization term to keep the learned policy within the scope of the target domain dataset. The effectiveness of OTDF is evaluated across various dynamics shift conditions with limited target domain data, demonstrating superior performance over strong baselines.", "questions": "1. Table 1 is too wide and not pleasing enough. Recommend using \\resizebox {\\linewidth} {!} to fix it.\n2. Why is the performance of IQL not assessed when using only source domain data?\n3. The paper does not provide the impact of using OT for data filtering or subsequent policy regularization on the performance bound of OTDF. A sub-optimal gap is expected to be given.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper introduces a novel approach to cross-domain offline RL by combining optimal transport with dataset regularization, which is a unique combination not commonly seen in the literature.\n2. The authors provide a solid theoretical foundation for their method, characterizing the performance bound of a policy and motivating their approach with theoretical insights", "weaknesses": "1. In previous studies, DARA, BOSA, and IGDF all utilize source and target domain data of similar quality, implying that the state space distributions of the datasets were closely aligned. This alignment ensures, to some extent, that there is a certain overlap between the source and target domain data. Your setting differs from theirs, which naturally raises the question of whether data filtering or policy regularization played the critical role. As the results in Figure 3 indicate, the introduction of policy regularization has a highly significant impact on the performance of the algorithm. However, in previous methods such as SRPO and IGDF, constraints related to this were not designed, which may account for the significant performance improvement observed in OTDF.\n\n2. In the paper \"Cross-domain imitation learning via optimal transport, ICLR 2022,\" some methods for cross-domain transfer based on optimal transport have already been adopted. What do you think are the differences between your method and the previous methods in terms of implementation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d3K489amGo", "reviewer": "Reviewer_vhAr", "summary": "This work proposes a new cross-domain offline RL method OTDF, which can leverage data from source domain to help the learning of the target domain. This work proposes that the core of the cross-domain offline RL is the dynamics mismatch between source and target domain, and introduces a threshold and a weight to only leverage those data close to the target domain in the source domain.", "questions": "Can a more complex kinematic shift be proposed?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This work proposes an informative proof and well-designed method. Also, this work can be a well baseline for follow-up works.", "weaknesses": "The $\\mu_{t,t'}$ is not defined in Eq. 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)