Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning
Authors: Ruhan Wang
Yu Yang
Zhishuai Liu
Dongruo Zhou
Pan Xu
Venue: Accepted by TMLR ()

Abstract: We study offline off-dynamics reinforcement learning (RL) to utilize data from an easily accessible source domain to enhance policy learning in a target domain with limited data. Our approach centers on return-conditioned supervised learning (RCSL), particularly focusing on Decision Transformer (DT) type frameworks, which can predict actions conditioned on desired return guidance and complete trajectory history. Previous works address the dynamics shift problem by augmenting the reward in the trajectory from the source domain to match the optimal trajectory in the target domain. However, this strategy can not be directly applicable in RCSL owing to (1) the unique form of the RCSL policy class, which explicitly depends on the return, and (2) the absence of a straightforward representation of the optimal trajectory distribution. We propose the Return Augmented (REAG) method for DT type frameworks, where we augment the return in the source domain by aligning its distribution with that in the target domain. We provide the theoretical analysis demonstrating that the RCSL policy learned from REAG achieves the same level of suboptimality as would be obtained without a dynamics shift. We introduce two practical implementations $REAG^{∗}_{Dara}$ and $REAG^{∗}_{MV}$ respectively. Thorough experiments on D4RL datasets and various DT-type baselines demonstrate that our methods consistently enhance the performance of DT type frameworks in off-dynamics RL.

Published in Transactions on Machine Learning Research (02/2026)
Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning
Ruhan Wang"
ruhwang edu
Indiana University
Yu Yang* Duke Universiby
yu..yong@tduke.co edu
Zhishuai Liu Duke University
liw@duke.ee..omo edu
Dongruo Zhou
dz19@iu. edu
Indiana University
Pan Xu Duke University
pen.zu@duke.cm edu
Reviewed on OpenReview: https: openreview. net/ forum? id- 0DV0r
Abstract
We study offline off-dynamics reinforcement learning (RL) to utilize data from an easily accessible source domain to enhance policy learning target domain with limited data. Our approach centers on return-condittoned supervised learning (RCSL), partieularly focus- ing Decision Transformer (DT) type frameworks, which can predict actions conditioned on desired return guidance and complete trajectory history. Previous works address the dynamics shift problem by augmenting the reward in the trajectory from the source domain to match the optimal trajectory in the target domain. However, this strategy can not be directly applicable in ROSL owing to (1) the unique form of the RCSL policy class, which explicitly depends on the return, and (2) the absence of straightforward representation of the optimal trajectory distribution. We propose the Return Augmented (REAG) method for DT type frameworks, where augment the return in the source domain by aligning its distribution with that in the larget domain. We provide the theoretical analysis demon- strating that the RCSL policy learned from REAG achieves the same level of suboptimality as would be obtained without a dynamics shift. We introduce two practical implementations REAG and REAGi respectively. Thorough experiments on D4RL datasets and vari- ous DT-type baselines demonstrate that our methods consistently enhance the performance of DT type frameworks in off-dynamics RL.
Introduction
Off-dynamics reinforcement learning (Eysenbach et al., 2020; Jiang et al., 2021; Liu et al., 2022; Liu Xu, 2024; Guo al., 2025) arises decision-making domains such as autonomous driving (Pan al., 2017) and medical treatment (Laber al., 2018; Liu al., 2023), where direct policy training through trial-and-error in the target environment is often costly, unethical, infeasible. A common strategy is to train the policy in source environments with similar but more accessible dynamics. However, discrepancies between the source and target environments create simulation-to-realit (sim-to-real) gap, which can lead to catastrophic failures when deploying the source-trained policy in the larget enviromment.
"Equal contribution.

---

Published in Transactions on Machine Learning Research (02/2026)
Beyond the challenge of dynamics shift, practical scenarios often do not allow real-time online interaction with the source environment due to time and computational constraints. As result, policies must be learned from pre-collected datasets generated by behavior policies. This setting is particularly difficult, as it combines off-policy, offline, and off-dynamics characteristics. Recently, supervised (earning-based methods (Chen et al., 2021; Brandfonbrener et al.; 2022) have emerged as more stable and scalable alternatives to traditional offline reinforcement learning algorithms grounded in dynamic programming (Levine et al., 2020). In the offline off-dynamics setting, the majority of training data is drawn from the source domain, with only limited portion collected from the target domain. Our study focuses on advancing Decision Transformer (DT) type frameworks (Chen et al., 2021; Hu et al., 2024; Zhuang et al., 2024) for off-dynamics reinforcement learning, which can be viewed as a special case of return-conditionel supervised learning (RCSL) (Emmons et al., 2021; Brandfonbrener et al., 2022). While DTpeype methods have gained siguificant attention across various reinforcement learning tasks, no prior work has explicitly tackled the off-dynamics RL problem.
There are several previous significant works in off-dynamics reinforcement learning that employ reward augmentation to address the dynamics shift. between source and larget environments (Eysenbach et al., 2020; Liu et al., 2022). In particular, Eysenbach et al. (2020) proposed the DARC algorithm to train a policy in the source domain using augmented rewards. These augmentations are derived by minimizing the KL distance between the distribution of trajectories generated by the learning policy in the source domain and those generated by the optimal policy in the target domain. Liu et al. (2022) extended this idea to the offline setting with the DARA algorithm. However, these reward augmentation techniques for dynamic programming based RL algorithms are not directly applicable to RCSL methods for two primary reasons. First, the policy classes used in RCST. methods explicitly depend on the conditional return-to-go function, leading to different trajectory distributions that invalidate the trajectory matching methods. Second, the augmentation techniques in Eysenbach et al. (2020); Liu et al. (2022) explicitly rely on the form of the optimal trajectory distribution in the target domain. In contrast, there is no straightforward representation of the optimal RCSL. policy and the trajectory distribution. Therefore, novel augmentation mechanisms must be derived for RCSL methods to effectively address off-dynamics reinforcement learning.
In this work, we propose the Return Augmented (REAG) algorithm, which augments the returns of trajec- tories from source environment to better align with the target environment within DT-type frameworks. Through rigorous analysis, we show that the RCSL policy learned with REAG in the source domain achieves suboptimality comparable to that of a policy learned directly in the target domain without dynamics shift. Building upon our preliminary study on return augmentation for Decision Transformer (RADT) (Wang et al., 2024), the present work substantially generalizes and extends this idea by developing a unified REAG frame- work that integrates augmentation across multiple DT-type architectures, introduces practical stabilization mechanisms, and provides expanded empirical evaluation under structured dynamics shifts. Specifically, our contributions are summarized as follows:
We propose novel method, REAG, specifically designed for DTT-type architectures in the off- off-dynamics RL setting. The approach augments the returns of offline trajectorics in the souree domain by leveraging a small amount of data from the target domain. We develop two practical implementations of REAG: REAG derived from reward augmentation techniques used in dynamic programming baaanmninn----ddddddddee methods, and RyAG direct return distribution matching.
We provide a rigorous thooretical analysis demonstrating that the return-conditioned policy learned via REAG can achieve suboptimality comparable to that of policy trained directly on the larget domain. Our analysis relies on the same data coverage assumptions as Brandfonbrener et al. (2022), which consider the no dynamics shift setting. This result implics that return augmentation can enhance the performance of RCSL in off-dynamics reinforcement learning, particularly when the available source dataset is sub- stantially larger than the target dataset..
We conduct experiments on the D4RL benchmark by training policies on source datasets collected from modified dynamics and evaluating them in the original environments. Across DT-type base- lines includin DT (Chen et al., 2021), Reinformer (Zhuang et al., 2024) and QT (Hu et al., 2024) both REAGiar and REAGTv consistently improve performance, with REAGTy showing the greatest gains, highlighting the advantage of return-level augmentation.
2

---

Published in Transactions on Machine Learning Research (02/2026)
2
Related Work
Off-dynamics reinforcement learning (RL) is a specialized domain adaptation problem that draws inspiration from broader Iransfer learning principles (Pan & Yang, 2009). Numerous algorit have been proposed to address the discrepancy between source and target environments (Niu et al., 2022; Liu et al., 2024; Xu et al., 2024; Gui et al., 2023; Lyu et al., 2024a;b). One promising approach involves modifying the reward function in the source domain. For instance, the DARC algorithm (Eyseubach et al., 2020) addresses this challenge in online settings by employing reward augmentation method that aligns the optimal trajectory distributions of the source and target domains. Building on this, DARA (Liu et al., 2022) uses reward augmentation to supplement a limited target dataset with a larger source dataset in offline scenarios. However, a known limitation is that policies learned in the source domain often require an additional imitation learning step to effectively transfer behavior to the target domain (Guo et al., 2025), Unlike these dynamic programming- based methods, our work adopts the adaptation setting of DARA but introduces a novel augmentation method specifically designed for Return- Condii-nd Supervised Learning (RCSL), with a particular focus the Decision Transformer.
Another category of methods utilizes source data directly without augmentation, instead applying learned filtering mechanisms based on dynamics mismatch. Specifically, H2O (Niu et al., 2022) performs importance weighting to penalize Q-values associated with large dynamics gaps in offline-toonllne transitions. VGDF (Xu et al., 2024) filters data based on value consistency in online scenarios, while CPD (Gui et al., 2023) utilizes a dynamics alignment. module to minimize discrepancies. Additionally, PAR (Tyu et al., 2024a) addresses off-dynamics issues by capturing representation mismatches. Recent benchmarks (Lyu et al,, 2024b) have also demonstrated that IQL maintains robust performance in off-dynamics settings. In the context of cross-domain offline RL, BOSA (Liu et al., 2024) tackles out-of-distribution (OOD) state-actions and dynamics through policy and value optimization, respectively. IGDF (Wen et al., 2024) selectively shares source domain transitions via contrastive learning, while SRPO (Xue et al., 2024) regularizes the policy using learned stationary state distributions. Most recently, MOBODY (Guo et al., 2026) proposed a model-based approach that learns proxy for the target transition model through joint representation learning, subsequently using generated target data for policy optimization.
Return Conditioned Supervised Learning (RCSL) has emerged powerful framework for offline RL (Brand- fonbrener et al., 2022). Notable implementations such as RvS (Emmons et al., 2021) and the Decision Trans- former (DT) (Chen et al., 2021) have demonstrated performance competitive with traditional RL methods. The core principle of RCSL is to condition policies on a desired target return. In this paper, we focus on the Decision Transformer, which treats offline RL as a sequence generation task. The strong generalization potential of DT has inspired its application in various scttings, including offlino-to-online RL. (Zheng et al., 2022) and me (Xu el al., 2022; Yang & Xu, 2025). However, to the best of our knowledge, no prior work has explicitly explored the adaptation capabilities of DT within the off-dynamics RL. framework.
3 Preliminary
Sequential Decision-Making. We consider a general sequential decision-making problem. At each step t, the agent reccives an obscrvation o from the environment. Based the history up to step t, the agent makes action a, and receives the reward ra from the environment. The agent interacts with the environment in episodes with a length H. We use == (01, "1, "", ,HH,HH, rH) to denote a whole trajectory, and we use g(r) = Di=" Pr to denote the cumulative return of the trajectory т. We model the environment Markov Decision Process (MDP) M, which consists of (S,A,,
Here S is the state space, each state
represents the possible history up to some time step t, i.c., s = (01, r1, ,0). A is the action space, p(s'|s, [s, a) is the transition dynamics that determines the transition probability for the agent to visit state s from current state s with the action a. r(s, a) denotes the reward function. We re-define a trajectory as == ($1,a1, a1, ... ,8H, ag TH)) We assume that cach corresponds to one single time step t= = t(s), and we denote gn (s) E,~z (g())s = s). Then the goal of the agent is to learn a policy π S + A that maximizes the expected accumulated reward J(T)= == [ We denote the optimal policy n².
3

---

Published in Transactions on Machine Learning Research (02/2026)
Offline RL and Decision Transformer. We consider the offline reinforcement learning setting. Given dataset D, the goal of the agent is to learn x² from D. We assume that the trajectories in DD are generated from a behavior policy 8. In this work, we mainly consider Decision Transformer (DT) (Chen et al., 2021) as our backbonc algorithm. DT is a type of sequential modeling tochnique based on Transformer (Vaswani et al.; 2017) to solve oflline RL problems. In detail, DT maintains a function T(a) 8, g) as its policy function. To train the policy n, DT aims to minimize the following negative log-likelihood function L(T) = (TT) == - -EED Eisigh log क (ar]st 9())). To evaluate n; DT defines conditioning function f:S + R. which maps each state to a return value and guides the policy xy within the environment, where (a) == raass, f(s)). The conditioning function is pivotal in DT, as varying f(s) for given state results in different policies. To achieve the optimal policy, f() should be maximized (Zhuang et al., 2024).
Offline Off-Dynamics RL. In this work, we consider the offline off-dynamics RL problem, where the agent has access to two offline datasets DS and DJ DS, DD' include the data collected from the source environment M and the target environment M The source and the target environments share the same reward function r, with different. transition dynamics ps and pf. In practice, we assume that. the dataset size from the source dataset [D$| is much larger than the data coming from the target dataset [D'). Then the agent aims to find the optimal policy for the target environment M based on the data from both the source and the target environments. Since the transition dynamics ps and pf are different, we can not directly apply existing RL algorithms on the union DS DD".
4 Return Augmentation for Goal Conditioned Supervised Learning
4.1 Return-Augmented Framework
DT has the potential to address offline off-dynamics reinforcement learning challenges, as shown in Table 1. However, it still has certain limitations. To overcome these, we propose a general framework that efficiently learns the optimal policy for the target environment using the combined dataset D U DJ. Leveraging the return-conditioning nature of DT, we introduce a return augmentation technique that modifies returns in the oflline source dataset through a transformation function. This approach allows the policy derived from the augmented source dataset to effectively approximate the optimal policy of the target environment, as illustrated in the following equation, where x represents a strong candidate for approximating the optimal policy of the target environment and v/ is the carefully chosen transformation function.
= arg mine L(x) == - EvEDs Eisigh log (ag ((((T))).
We call our method Return Augmented DT (REAG). Next we introduce two methods to construct vb based on the dynamics-awary reward augmentation (DARA) technique (Eysenbach et al., 2020; Liu et al., 2022), and a direct return distribution matching method.
4.2 Synamics-Aware Reward Augmentation
We first summarize the idea of dynamics-aware reward augmentation (DARA) (Eysenbach et al., 2020; Liu et al., 2022). Let (s'1s, a) denote the transition dynamics of the larget environment, and 1 (s'1s, a) denote the source environment. According to the connection of RI. and probabilistic inference (Levine, 2018), we can turn the optimal policy finding problem into an inference problem. We use O to denote a binary random variable where O 1 suggests T is a trajectory induced by the optimal policy. Given a trajectory T, the likelihood of 7 being a trajectory induced by the optimal policy under the target environment is pT(0- 1/r) = exp(E r(st, a4)/n), where 7) is the step size parameter used for tuning, It means that the trajectory with higher cumulative rewards is more likely to be the trajectory induced by the optimal policy. We introduce variational distribution (T) = P(s1) I[ pS (st+1158, az (alls))): to approximate (= ((001/7] 1/7). Then we have
log PT (O- = 1) = log EY (T()) PT (0 1/T)
EyWPS(() loog p0(00= (0 1|7) + log (Pr (T)
Enwps()) r /n log (t) ai))],
(4.1)
4

---

Published in Transactions on Machine Learning Research (02/2026)
where for the first inequality, we change the distribution of the expectation from (T) to (T) and then use Jensen's 's inequality to derive the result; the second equation holds due to the assumption /modelioggggggggg that the likelihood of r being a trajectory induced by the optimal policy under the target environment is PT = 1/7) = exp(Ei=1 r(st, a))/n). Therefore, we obtain an evidence lower bound of ( = 1), which equals to find a policy to maximize the value in the source environment, with the augmented reward o (st, r(st, at) + 17 PT (st+1 [st, a+) 1ogg P (st+11 (st,)))) Following Eysenbach et al. (2020), to log (sc+ az), we use a pair of learned binary classifiers which infers whether the transitions come from the source or target environments. Specifically, we denote classifiers lsao (-18, s') and qsa (-1s, a), which return the probability for some (s, or (s, a) tuples whether they belong to the source or the target environments. Then according to Eysenbach et al. (2020), we have
az).
estimate the log Pd (5t+118t,a,))
log pd (8 (86,))8,,0)) 0z) log pS (8441/81,00 ae) = Ar(st; (t: 8t+1)
g(M" 1t,, at, S1+1)
Isa gea (Ma(M" Stc as)
:= log g(M 18,, G2 S1+1)
log Isa ( (M 18c, at)
(4.2)
For a trajectory T = ($1, 01, .i, 8H, ag, TH); we denote the transformation (g((s))) = Enne Th +
Eie (htt1)). We denole such transformation method REAG
4.3 Direct Matching of Return Distributions
The reward augmentation strategy in REAG stems from the probabilistic inference view of RL which matches the distribution of the learning trajectory in the source domain with that of the optimal trajectory in the target domain (Eysenbach et al., 2020). However, it does not fully capture the power of DT, which is able to induce family of policies that are conditioned on the return-to-go f. By varying f, DT enables the generation of diverse range of policies, including the optimal one. In contrast, REAG assumes single, fixed target policy, and thus its augmentation strategy cannot generalize across multiple policies induced by varying f in DT. As a result, it cannot find the desired return conditioned policy when evaluated with a different f in the target domain. This motivates us to find a return transformation method te to guarantec that (a(ss) as (als) for all f.
We consider a simplified case where both D and D are generated by following the same behavior policy (()ss). We use ds(A) and (A) to denote the probability for event A to happen under the source and target environments following B. With a slight abuse of notation, we use gs and gr to denote the return following the behavior policy. Denote (s(s) and (r (gs(s, a) and gr(s, a), resp.) as the random variables representing the relurn-to-go starting from 8 ((s,a); resp.) in the source and target domain. Then we characterize the learned policies by DT under the infinite data regime (Brandfonbrener et al., 2022) for both the souree environment and target environment. According to Brandfonbrener et al. (2022), ザÿ (als) = ps (als, (gs)) f())). Then we can express S and -- as
(a(ss)) ds(als)ds(((gs) = = (s)) ds(b(gs) = [()))s)
(Fl())) = dr(als)dr(gr = f()))s,a)
dr(er = [(8) 8)
Since the behavior policies over the source and target environments are the same, we have ds(a)s) = dr(ass) for all (s. Then in order to guarantee TT = (a]ss)) we only need to guarantce ds(#(gs(s,, a)) -|s, a) = ()) = -(s,a), Vs, a. Denote the cumulative distribution function (CDF) of gs conditioned on 8, a is gs|s, a ~ G (s,a), and gr|s, a ~ (), Then if both (s, and G7 a) are invertible, we can set sb as follows
(lgs) = (G
(4.3)
If there exist ps, pr, and r such that the DARA-type augmented reward-to-go satisfies (4.3), then the DARA-type reward augmentation can be deemed as a special case of the transformation (4.3). In general, of and G. are hard to obtain and computationally intractable, making & intractable either. We use Laplace ap- proximation to approximate both G. and G3 by Gaussian distributions, e.g., () N(2 a), ())
5

---

Published in Transactions on Machine Learning Research (02/2026)
and (3(s,a)~
Then it is easy to obtain that
(gs) == Ds - o as
+ +
(4.4)
の (s,,))
We denote DT with a & transformation from (1.4) by REAG since such transformation only depends on the estimation of mean values nº, n² and variance of, of.
4.4 Sample Complexity of Off-Dynamics RCSL
In this section, we provide finite sample analysis of the sample complexity of the off-dynamics RCSL. To this end, we first define some useful notations. We assume there are 8 trajectorics in the source dataset D8, and N trajectories in the target dataset D Denote P the joint distribution of state, action, reward and return-to-go induced by the behavior policy 3 in the source environment, and PJ in the target environment. Denote de as the marginal distribution of state induced by any policy in the source environment, and df in the target environment.
Denote (xx as the expected cumulative reward under any policy æ and the target environment. For any return-to-go g in the source dataset DS we transform & by an oracle defined in (4.3) with others remain the same, then we get modified dataset D We denote the mixed dataset as D = DJ UD$. Our theorem is established based on the following assumptions.
Assumption 4.1. (1) (Near determinism) (r(rr & r(s, a) or s' * E(s,a)|s,a) s e at all s,a for some functions& & and r. (2) (Consistency of f) f(s) f(s') +r for all s. Assumption 4.2. For all s we assume (1) (Bounded occupancy mismatch) P (s) s Cypp (2) (Return coverage) P3 (g f(s)]s)= 2 ax; and (3) (Domain occupancy overlap) (f)) sdd((s).
Assumption 4.3. (1) The policy class IT is finite. (og - log x (a'ls' I for any (a,s,g,"', ".g,,',,''g) and all E II. (3) The approximation error is bounded by eapprox: i.e. min L(T) s Capprox-
Assumptions 4.1 to 4.3 are the same as the assumptions imposed Theorem 1, Theorem 2, and Corollary in Brandfonbrener al. (2022) respectively.
We first show the sample complexity of DT with only the samples from the target dataset DT. If we only use the offline dataset D' collect from the target environment, i.e., at training time we minimizes the empirical negative log-likelitee lieliikl loss:
(T(T)= = Σ
log (ag
TED IStSH
Then we get the following sample complexity guarantoc basod on the result in Brandfonbrener et al. (2022). Corollary 4.4. There exists a conditioning function f S R such that assumptions (1) and (2) in Assumption 4.1, and (1) and (2) in Assumption 4.2 hold. Further suppose Assumption 4.3 holds. Then for some 8 E (0, 1), with probability at least. - ō, we have
J²(**)- J"(y)) < ( (V
(lo log |11/58 1/4 + [eapprox) NT
E (22) as
af
Now we consider the case of mixed dataset, where we train our policy on both the target dataset and the source dataset using the proposed returned conditioned decision transformer methods. Note that the size of the target enviromment dataset is usually small, while the size of the source environment dataset is much larger, that is, NT & S If we incorporate the modified source dataset into the supervised learning, that is, we minimize the following empirical negative ligeeiiol loss:
i mixx mm) = Σ
Σ
log log g(x)))
(4.5)
TED 1S1SH
6

---

Published in Transactions on Machine Learning Research (02/2026)
An observation is that, with the modified source dataset, the regret J" (T() JT(tt) can be significantly reduced. This is formulated in the following theorem.
Theorem 4.5. There exists conditioning function such that Assumptions 4.1 and 4.2 hold. Further assume Assumption 4.3 holds. Then for some å E (0, 1), with probability at least 1 - 8, we have
N + - o ( af / NT H² (vo log 111/8 1/4
+ v Eappiox) + _ -12. (4.6)
N + NS
+N3
Remark 4.6. Compared to Corollary 4.4, Theorem 4.5 suggests that the modified samples from the source domain could enhance the performance of RCSL when the domain occupancy overlap coelficient Y is large. In particular, when N > NT and 7/ = 0(), (4.6) can be simplified to
J < o Ci H2 (Vo log 11/66 1/4 af
+ + VEapprox) + _H²) af
E
N
which significantly improves the bound on suboptimality in Corollary 4.4.
5
Experiments
In this section, we first outline the fundamental setup of the experiment. We then describe experiments designed to address specific questions, with each question and its corresponding answer detailed in separate subsection.
How effective are DT-type methods in mitigating the impact of oflline data shortages in target environ- ment?
What techniques can be employed to improve the performance of DT--ype methods in off-dynamics scenarios while addressing the constraints of offiine data shortages in target environment?
How does the performance of DT-type methods compare other approaches in solving off-dynamics problems?
5.1 Basic Experiment Setting
Tasks and Environments. We study established D4RL tasks in the Gym-MuJoCo environment (Fu et al., 2020), a suite built atop the MuJoCo physics simulator, featuring lasks such as locomotion and manipulation. Particularly, we focused on three environments: Walker2D,, Hopper, and HalfCheetah. In addressing the off- dynamics reinforcement learning problem, we distinguish between the Source and Target environments. The Target environment is based on the original Gym-MuJoCo framework, while the Source environment is modified using two shift methods: BodyMass Shift and JointNoise Shift. In the BodyMass Shift, the mass of the body is altered in the Source environment, whereas in the Noint Shift, random noise is added to the actions.
Dataset. For the Target Dataset corresponding to the Target Eavironment, we leverage the official D4RL data to construct the target datasets: 10T and IT. The 10T dataset comprises ten times the number of trajectories compared to the 1T dataset. For the Source Dataset collection, we begin by modifying the environment through adjustments to the XML file of the MuJoCo simulator. We then collect the Random, Medium, Medium- Replly, and Medium -xpeert offline datasets in the modified environments, following the same data collection procedure as used in D4RL. For further details on the dataset collection process and the datasets, please refer to the Appendix B.
Basclines. We consider a diverse set of established off-dynamics RL methods as basclines, including BEAR (Kumar et al., 2019), AWR (Peng al., 2019), BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020), and MOPO (Yu al., 2020). We further include their DARA-angmented variants (Liu et al., 2022) additional baselines for comparison. In addition, evaluate recent dynamics-aware methods, including
Unlike the approach of Liu et (2022), which constructs the dataset by selecting the last 1/10 timesteps from the original target dataset (1OT), we propose uniform sampling method across trajectories the target dataset.
7

---

Published in Transactions on Machine Learning Research (02/2026)
BEAR
AWR
BCQ
CQL
M
M-E
M
M-R
M-R
M-E
M
M-E
1T
1.63813.90
8.77740085
68,023.1
28 um
100, 360.566.050 sia
62.567.a 80.638 cen
101.600-1am
57.402 65
1111611 iara
10T
13. H3gsaus
6.8221124
21 21.37
78000
38 280g1144
154gamm
74.736ggass as
64.735
101 100
78. 19120
40.14
101 101.80
MOPO
DT
Reinformer
QT
M
M-R
M-F
M
M-R
M-E
M
M-R
M-F
M
M-R
M-F
20:9534275
40385 27.251
34 4454820
171
29,07441 t00
38.07227u4 101,254g5.n
81.758
67,54649.0 111.72241.8
10T 22 1622: 18,3291199 21 19612993
79.69713314
11352811101
108 108.922.1
8133T11 08.10812m0
109.815
58. $8.262.11.86 122.05
$5,092.x.7
111.301:.mm
Table I: Performance comparison of algorithmus on the 1T. 10T, and 1T10S datasets. In this study, (T10S(B) refers to the source dataset under the Mody shift setting, while 1T10S(J) corresponds to the source dataset under the JointNoise shift setting, Experiments are conducted using the Medium (M), Medium-Replay (M-R), and Eeepiut-Expert (M-E) datasets. We present the results for the Walker2D environment here; complete results are provided in Appendix C.I. All reported values are averaged over five seeds (0, 1012, 2024, 3036, 4048).
DFDT (Wang et al., 2025), H2O (Niu et al., 2022), and IGDF (Wen et al., 2024). hyperparameter selec- tion, we use consistent scttings across tasks for shared paramcters, such as the learning rate and the number of training iterations. Detailed configurations are provided in Appendix B. parameter settings.
5.2 Evaluation of Adaptability and Data Efficiency in RCSL Algorithms
We evaluate three representative DT-type algorithms include DT (Chen et al., 2021), QT (Hu et al., 2024), and Reinformer (Zhuang et al., 2024) to assess their ability to enable an adaptive policy while reducing reliance on offline data in the target environment. To conduct this evaluation, we perform two experiments: (1) We examine the performance of the three DT-ttpe algorithms under varying dataset sizes and quality levels in the target environment: (2) We evaluate their effectiveness in off-dynamics scenarios.
To assess the impact of dataset size and quality on the performance of DT-type algorithms, we evaluate three algorithms using two datasets: a subset of the target data (1T) and the full target dataset (10T), comparing the results against other baselines. These experiments aim to quantify the performance gap between training on 1T and 10T datasels, highlighting the effects of target environment data scarcity and establishing a benchmark for off-dynamics settings. In off-dynamics offline RL, instead of relying solely on large target dataset, we incorporate a small subset of target data with a larger source dataset. To examine how effectively algorithms leverage source data, we construct the 1T10S dataset by combining a subset of target data (1T) with the full source dataset (10S), following the setting of Liu et al. (2022). This dataset serves as the training set for DTT-type algorithms, whose performance is then evaluated in the target environment. For a comprehensive comparison, we benchmark DT-type algorithms against other bascline methods.
Table 1 presents the evaluation results, illustrating the impact of dataset size and off-dynamics settings on algorithm performance. Limited training data constrains the algorithm's learning capacily, leading to degraded performance, particularly when the target environment data is scarce. To mitigate this issue, additional source datasets are incorporated under BodyMass Shift and JointNoise Shift settings, improving generalization to the target environment. While leveraging source data can partially compensate for the lack of target data, it remains suboptimal compared to training with sufficient target environment data. To enhance the effectiveness of DT--type frameworks in off-dynamics settings, we propose two return-based augmentation methods: REAG and REAG which can be applied DT, Reinformer, and QT frame- works. Specifically, applying REAGE results in REAGDT R.AGR and REAG MV: while applying REAG leads to REAGDT REAGBain Reinf and p demonstrating the potential of these aug- mentation techniques in improving algorithm performance under off-dynamics setting.
5.3 Return Augmentation Methods for Off-Dynamics RL
Here we discuss how to implement REAG and REAG, in practice. We implement REAG based on the dynamics-aware reward augmentation method proposed in Liu al. (2022). For REAG it involves training the CQL model across both the Target and Source Environments to derive the respective value
8

---

Published in Transactions on Machine Learning Research (02/2026)
DT
Reinformer
QT
1T108
REAGEY
REAG
1T108
REAGRY
REAG
IT10S
REAGY
REAGE
BM
28.768=1223
$0.87711.151
282
80857=00.00
80.566
84.325661425
84.582+0.507
$3.069+0.850
M
JN
71.068410022
7007
74.488117211
75.08800.999 980
74. 31
80.21411143
1.90
28.67242.211
W2D M-R
BM
73:00+119200
3377
67.365507790
7000
07202257777
10.29614.2111
06.55844.3055
87. 87.29240:631
87,491+1 226
76.169
JN
08:20613.181
56.72242.663
02:22:103385
51,0113::177
47.091110.294.
00.8381-18831
82,13911.029
82.3631 1,20
79.79614.708
81:400008233
88 88.235=1..881
85.52840.8631
83.088
81.87711.117
83.76140.7551
93.082400.348
92.746 199.
1.577
M-E
116/7462:111 116 1110000:2:277
117.30012.000
118:21811-601
117.705::2 199
116.14911.660
118
118.6011000877 6
116.115118888
BM
01:057=0.117
39.435:112997
37.787=1.314
51.35799.713
59.08542.7911
51.77145:12221
49.51649.7995 798
51.79649.9711
62.26265.348T
M
70.065:077755
78.35:22.221
70.3004-.633
72.36155877
70.8601:77281
08, 08.06617.077
73.98718.0000 080
708:12912.10 160
1Ip
M-R
BM
5.04
06 06.00243:2331
60.30==1116..
17.531
20.35249.79c
12573
63.963413.9948
#2.786 76.26747..1107
1922
JN
G1
018800020
77 875-2:11110 133
1.288
41.886156773
43.98955507571
52.052410.085
93.706377.50 559
03.400244.0000
51. 12.166611.111 168
BM 33.53110.866
42
33.63111.615
68.973
14.2111121177.
73.36347.674f
61.16244.767
77.270 16.092116.901
6071
M-E
JN 10S 106-254:1100 583
300367110000 0841
10::0122011
100:296::0130
100/47240100 1031
109.266:0.1888.
2000666:02114
109 109.803+0.0009
109 7.1
M RM
40220+0-911
37.000003390
37-333+0483
42.41100000
38/61+1:2381
44.896+0.643
47.303++.3188
46 068885
44 44.19:36722
47:833002841
48274420191
13.00900.007,
48 10-8001001081
56,21340.327
52.399411 4131
56 56.02610:4104 4100
IIc
M-R
BM 20 20696699.677
27.32233.250
2603042.271
31.58111.248
32.11441:4451
26.93514.375
41.900411.77
42.0155117729
41.15940.9651
0630044414
38.17:14.065
3801133321
4020042.014
4088042880
53 63,26320.73
53.80000.811
53.261+0.686.
BM 51.911111177
36.22822.3011
51.35748.231.
10.56820:981
$5.818&1899 46.08811.657
71.08048.802
69.81945.1201
70.53348.0221
M-E
JN 70.37338.000 00770:220000
72:75122.027
76,03438888
79.300001440
18.081+1.198
82,9614-4.119
83.60240.009
$2,14842788.
Table 2: Performance evaluation of two return augmentation methods, REAGMV and REAG Dara integrated with DT, Reinformer, and QT frameworkes in off-dynamics scenarios. The experiments conducted in the Walker2D (W2D), Hopper (Hp), and HalfCheetah (Hc) environments under the Medium (M), Replly (M-R), and Medium- Expert (M-E) settings. The source enviromment is modified using two shift conditions: BodyMass shift (BM) and Joint shift (JN). For reference, the table also includes the performance of the original DT-type methods without augmentation, displayed in gray font. Performance changes due augmentation are indicated with red upward arrows (1) for improvements and green downward arrows (J) for degradations compared to the original DTT-type methads. All reported values are averaged over five random seeds (0, 1012, 2024, 3036, 4048).
functions, denoted as Qr and Qs- The derived value functions are then used to relabel the returns of trajectorics in the original datasct. More specifically, the relabeled return g is calculated as defined in (4.4). Within this framework, we use (x, to denote Qs(s,a), a), and Qr(s, a) corresponds to y² (s, a). For the computation of as(ss,) a) and or(s,a), we employ the following methodology: For a given state s, we use the policy of CQL on the source datasct to obtain n available actions (af, aa....,,a,, given the state s, with the corresponding Q values (Qs(s,af) (s (ssss,)), ag),... ....(s((s,,))) and n available actions {af, ab......?") in the target environment obtained from the CQT. policy trained over the target dataset, with the corresponding Q values {Qr(s, a."), (r(s,a))) a.))....,Q Qr(s, o²)} The standard deviations as(s, and (s, a) are then calculated as specified as follows.
as(s, a) = std(Qs Qss,s Qs(s, ag)). ar(s,a) = std(Qr a Qr(s, , Qr(s,
For a detailed discussion, please refer to Section 4. As defined in (4.4), computing the ratio an(s.a) asts,a] is essential. However, extreme values of this ratio can lead to instability during training. To address this, we introduce a clipping technique that constrains the ratio within an upper bound 0, and a lower bound 0g. This helps stabilize REAGMY training by mitigating two key challenges. First, since this ratio depends on the variance of return-to-go in both the source and target environments, extreme variance values can introduce large gradients or noisy updates, destabilizing training. Sccond, variance is estimated using the Q-value function learned through CQL on the source and larget datasets, which may introduce estimation errors. By bounding the ratio within controlled range, clipping reduces the impact of these errors and prevents instability.
Table 2 presents a performance comparison of the REAG and REAG return augmentation techniques integrated into different DTT-type frameworks, inchuding DT, Reinformer, and QT, in off-dynamics scenarios. Evaluations are conducted across three Gym environments- D Hopper, and HalfChectah- under three dataset settings: Medium, Mepium-Replay, and Kedium-Expert To simulate off- o--dynamies conditions, two domain shifts, Masy Shift. and Noise Shift, are introduced. The results demonstrate that
9

---

Published in Transactions on Machine Learning Research (02/2026)
Baselines
DARA Baselines
DT-type
REAC
REAGHY
25
RankScore better)
21.17
20
14.06
15.44
10-
10.89
7.33
7.72
9.94
10.07
12:44
12.11
11.06
11.06
5-
4.06
BEAR
AWR
BCQ
COL
MOPO
H20
1GDF
DFDT
D-BEAR
D-AWR
D-BCO
D-COL
D-MOPO
Reinformer
DT
REAGOW
REAG
REAG
RE REAGE
REA REAGIN
REAGET
Figure 1: Average normalized rank scores for all baseline algorithms across the Medium, Medium Medium-Replay, Replay, and Medium Medium-Expert datascts under BodyMass and JointNoise shift settings in the Walker2D, Hopper, and HalfCheetah environments. Within cach setting, algorithms were ranked based performance, with the top-performing algorithm assigned a rank of 1. Tied scores received the same rank, with subsequent ranks adjusted accordingly. Lower rank scores indicate better overall performance. The original ranks (from 22 algorithms) were normalized scale of 1 to 22. The figure presents the average normalized rank the Walker2D, Hopper, and HallCheetah environments.
1T10S
REAGMY
REAG
B-0.735
B-1.47
B-2.205
80
85
80
70
60
80
75
50
75
70
DT
Reinformer
QT
DT
Reinformer
QT
DT
Reinformer
QT
(a) Performance evaluation under varying body mass shift settings in the Walker2D medium environment.
1T10S B-2.50
REAG
REAG
B-1.25
B-3.75
65
60
95
50
50
35
40
85
20
30
80
DT
Reinformer
QT
DT
Reinformer
QT
DT
Reinformer
QT
(b) Performance evaluation under varying body shift settings the Hopper medium environment.
Figure 2: Performance of REAG and REAG algorithms under different body shift settings in the Walker2D and Hopper modium environments. "B-x" denotes that the body mass in the simulator is The target body is 2.94 the Walker2D environment and 5 in the Hopper environment.
both REAGHY and REAG effectively enhance DT-typpe frameworks, improving performance most off- dynamics scenarios compared to their original, non-augmented counterparts. Specifically, REAGHy which augments based return values, leverages information from both the source and target environments, making it particularly well-suited for return-based algorithms. In contrast, REAG which augments based on reward values, exhibits more variable performance across different environments and dataset set- tings. While REAG improves performance in certain cases, REAGLy delivers more stable and robust improvements the most cases. DARA is a widely adopted approach for addressing off-dynamics RL prob- lems by introducing reward augmentation to enhance policy adaptation from a source dataset to target environment while minimizing reliance on extensive target data. It seamlessly integrates with traditional offline RL frameworks such as CQL and BCQ. In our evaluation, compare proposed methods against DARA-based approaches, including both traditional RL frameworks and their DARA-augmentee variants, as well as DT-type frameworks with and without REAG and REAG augmentation, providing com- prehensive assessment of augmentation techniques for ofl-dynamics adaplation. We present a comparative ranking where lower average rank scores indicate better overall performance, as shown in Figure 1; for the raw results of cach setting, please refer to Appendix C.1. The results demonstrate that DT-type frameworks exhibit strong potential in solving off-dynamics RL problems, outperforming traditional offline RL methods, particularly in the case of QT. Return-based augmentation techniques further enhance effectiveness, with
10

---

Published in Transactions on Machine Learning Research (02/2026)
1T10s J-0.05
REAGHY
REAG J-0.5
1T10s J-0.05
REAGHY
REAG
J-0.5
80
80
80
90
70
70
70
50
60
50
60
30
DT Reinformer
QT
DT
Reinformer QT
DT Reinformer QT
DT
Reinformer QT
(a) Performance evaluation under varying joint noise shift, tings the Walker2D medium environment.
(b) Performance evaluation under varying joint noise shift set- tings the Hopper medium environment.
Figure 3: Performance of REAG My and REAG algorithms across varying JointNoise shift settings the Walker2D and Hopper medium environments. "J-x" denotes the addition of random noise the range (-x, +x) the action.
REAGM and REAGM achieving slate-of-the-arl performance compared to other baselines. Additionally, while DARA effectively improves the performance of men-reeurn-basee offline RL methods, a noticeable gap remains between these approaches and DT-type methods.
5.4 Ablation Studies for Return Augmentation Methods
In this section, we present an ablation study identify the key factors affecting REAG and REAGT algorithm performance. We study the following variation settings.
(Dynamics Shift) How do REAGMy and REAG methods perform under significantly shifted source enviromments?
(Clipped Augmented Return) What is the impact of the clipping technique in REAG on its performance?
(Consistent Augmented Return) How does consistency affect the performance of REAG
(Return Learning) How accurate is the estimation of the learned value function in REAG
Dynamics Shift. To evaluate the impact of shifting source environments on REAGMv and REAG we assess their performance under various BodyMass and Joint shift settings. The experimental results are presented in Figures 2 and 3. Our findings indicate that as the body mass shift increases creating greater discrepancy from the target enviromment performance deteriorates in both the Walker2D and Hopper Medium environments. Similarly, introducing higher levels of action noise leads to decline in performance, suggesting that increased random noise raiscs the likelihood of failure, ultimately resulting in poorer outcomes. This performance degradation is particularly evident in the DT framework, highlighting its sensitivity to off-dynamics shifts, whereas Reinformer and QT demonstrate greater robustness. Across all shift experiments, REAGHy consistently outperforms REAGiara with the performance gap becoming espocially pronounced under larger shifts, such as in the Hopper environment with a body mass shift of 1.25.
Clipped Augmented Return. For data augmentation in REAG we apply a clipping technique to prevent the occurrence of extreme values. To evaluate its impact, compare the performance of REAGr with and without clipping in the Walker2D, Hopper, and HalfCheetah Medium Expert environments under BodyMass shifts. The results, presented in Figure 4, demonstrate that mitigating extreme values generally enhances the performance REAG Additionally, we observe that for REAG clipping docs not yicld significant. improvements compared to DT and Reinformer. We hypothesize that this is due the QT mechanism, which inherently regularizes the return, whereas DT and Reinformer lack such a mechanism.
Consistent Augmented Return. It is worth noting that augmented target returns do not satisfy the consistency condition, which requires that the augmented returns follow R,41 Rg Tt, enforced by the original DT. To verify whether consistency is necessary condition for augmentation in off-dynamics settings, we conduct the following ablation study. Specifically, we introduce a variant of REAG denoted as REAGT (consistent), where for each trajectory the target environment, return augmentation is ap- plied only the initial return, while all subsequent augmented returns are derived using the consistency condition R(+1- R, rt. The results, presented in Figure 5, indicate that REAGHy outperforms its consistency-enforeed variant in most cases. This finding suggests that enforcing consistency does not neces-
11

---

Published in Transactions on Machine Learning Research (02/2026)
MEAGMV
REAGi (w/o clip)
Walker2D
Hopper
HalfCheetah
95
80
80
90
60
60
85
40
40
80
20
20
DT
Reinformer
QT
DT
Reinformer
QT
DT
Reinformer
QT
Figure 4: Comparison of REAGTV Mv with and without the clipping technique in the Medium Expert setting of the Walker2D environment under BodyMass shift. Results are averaged over five seeds.
REAGiv
Walker2D __M
Walker2D MR
Walker2D ME
85
80
95
80
60
90
75
40
85
70
20
80
DT
Reinformer
QT
DT
Reinformer
QT
DT
Reinformer
QT
Figure 5: Comparison of REAGTY and REAG My (consistent) across Medium, Medium Heplay, and Medium Expert settings in the Walker2D environment under BodyMass shift. Results are averaged over five seeds.
Cumulative Rewards
Cumulative Rewards
4000
Trajectory Return
Trajectory Return Value
2000
Value
3000
2000
1000
200
400
600
800
1000
200
400
600
800
1000
Timestep
Timestep
(a) Comparison between cumulative rewards and estimated Q5 values in the environment with 100 trajectories.
(b) Comparison between cumulative rewards and estimated
QT values in the target environment with 100 trajectories.
Figure 6: Comparison of the cumulative returns and the learned Q-values for the source (left) and target (right) environments
using CQL. Results plotted with the and variance of 100 trajectories.
sarily improve performance; instcad, it may limit the effectiveness of REAG in the context of off-dynamics offline reinforcement learning.
Return Learning. To evaluate the learned value functions, Qs and Qr, and their impact REAG conduct an ablation experiment. Specifically, assess the quality of the learned value functions in both the source and target domains. We select the Hopper environment with a medium-expert offline dataset as the target domain and the BodyMass shift as the source domain. Idcally, the value functions Qs and Qr learned through REAGY should accurately reflect the returns of trajectories in their respective domains. To verify this, we train two additional DTs separately on the source and target offline datasets obtain policies for these environments. Using these policies, we generate test trajectories through rollouts and then leverage the learned value functions Qs and Qr, trained on the 10S and 1T datasets, to predict the returns of these test trajectories. By comparing the predicted returns with the actual returns, assess the accuracy of the learned value functions. As shown in Figure 6, our learned value functions Qs and Qr accurately reflect the returns of trajectories collected by the policies in the source and target environments, demonstrating that the valauus used in our approach serve as reliable approximations.
6 Conclusion and Future Work
We introduced the Return Augmented (REAG) method for improving the performance of decision trans- former type of methods in off-dynamics reinforcement. learning which augments the return function of trajec-
12

---

Published in Transactions on Machine Learning Research (02/2026)
tories from the source environment to better align with the target environment. We presented two practical implementations: REAGDara' derived from existing dynamic programming reward augmentation techniques, and REAGMv: which matches the mean and variance of the return function under a Gaussian approximation of the return distribution. Through rigorous theoretical analysis, we showed that REAG when trained only on the source domain's 's dataset can achieve the same level of suboptimality as policies learned directly the target domaín. Our experiments demonstrate that REAG boosts various DT-type baseline algorithms in off-dynamics RL, and outperforms many dynamic programming based algorithms in this setting. This work establishes REAG promising method for leveraging source domain data to improve policy learning in target domains with limited data, thus addressing key challenges in offline, off-policy, and off-dynamics RL. Future work could explore extensions and applications of REAG in more diverse RL environments and further refine the augmentation techniques to enhance efficiency and effectiveness.
Acknowledgments
We would like thank the anonymous reviewers and the editor for their helpful comments. Yu Yang, Zhishuai Liu and Pan Xu were supported in part by the National Science Foundation (DMS-2323112) and the Whitehead Scholars Program at the Duke University School of Medicine. The views and conclusions in this paper are those of the authors and should not be interpreted representing any funding agency,
References
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? Advances in Neural Infor- mation Processing Systems, 35:1542 1553, 2022. 2, 3, 5, 6, 16, 23, 26
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-1509,, 2021. 2, 3, 4, 8, 18
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751, 2021. 2,
Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement lcarning: Training for transfer with domain classifiers. arXiv preprint arXiv:2006.1900 13916, 2020. 1, 2, 3,
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2063 PMLR, 2019. 7, 18
Haiyuan Gui, Shanchen Pang, Shihang Yu, Sibo Qiao, Yufeng Qi, Xiao He, Min Wang, and Xue Zhai. Cross-domain policy adaptation with dynamics alignment. Neural Networks, 167:104-117, 2023.
Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, and Anqi Liu. Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation. Advances in Neural Information Processing Systems, 37:136326 136360, 2025. 1,
Yihong Guo, Yu Yang, Pan Xu, and Anqi Liu. MOBODY: Model-based ofl-dynamics ofllnne reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openrevvew.ne/eeeeeeeeee httt///oroo//dd///70333omm
3
Shengehao Hu, Ziqing Fan, Chaoqin Huang, Li Shen, Ya Zhang, Yanfeng Wang, and Dacheng Tao. Q-value regularized transformer for offline reinforcement. learning. arXiv preprint or orXin:2405. 7098, 2024. 2, 8, 18
13

---

Published in Transactions on Machine Learning Research (02/2026)
Yifeng Jiang, Tingnan Zhang, Daniel Ho, Yunfei Bai, C Karen Liu, Sergey Levine, and Jie Tan. Simgan: Hybrid simulator identification for domain adaptation via adversarial reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2884 2890. LEEE, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019. 7, 18
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Newral Information Processing Systems, 33:1179 1191, 2020. 7, 18
Eric B Laber, Nick J Meyer, Brian J Reich, Krishna Pacifici, Jaime Collazo, and John M Drake, Optimal treatment allocations in space and time for on-line control of emerging infectious disease. Journal of the Royal Statistical Society Series C: Applied Statistics, 67(4))743 789, 2018.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint Xrv: 00909, 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprinl arXi::2005006443, 2020. 2
Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-aware reward angmentation in offline reinforcement learning- arXio preprint or wxxx:222330060 066662 2022. 1, 2, 3, 4, 7, 8, 17, 18
Jinxin Liu, Ziqi Zhang, Zhenyu Wei, Zifeng Zhuang, Yachen Kang, Sibo Gai, and Donglin Wang. Beyond ood state actions: Supported cross-domain offline reinforcement learning. In Proceedings of the AAAI
Conference on Artificial Intelligence, volume 38, pp. 13945-13953, 2024. 3
Zhishuai Liu and Pan Xu. Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation. In International Conference on Artificial Intelligence and Statistics, PMLR, 2024.
pp.
Zhishuai Liu, Jesse Clifton, Eric B Laber, John Drake, and Ethan X Fang. Deep spatial q-learning for infections discase control. Journal of Agricultural, Biological and Environmental Statistics, (844)774 2023.1
Jiafei Lyu, Chenjia Bai, Jing- Wen Yang, Zongqing Lu, and Xiu Li. Cross-domain policy adaptation by capturing representation mismatch. In International Conference on Machine Learning, pp. 33638-33660 PMLR, 2024a. 3
Jiafei Lyu, Kang Xu, Jiacheng Xu, Mengbei Yan, Jingwen Yang, Zongzhang Zhang, Chenjia Bai, Zongqing Lu, and Xiu Li. Odrl: A benchmark for off-dynamics reinforcement learning. arXio preprinl arXiv:2410.20m0 2024b. 3
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When to trust your sim- ulator: Dynamics-awar aware hybrid affline-and-onlee reinforcement learning. Advances in Neural Information Processing Systems, 35:36599 36612, 2022. 3, 8, 18
Sinno Jialin Pan and Qiang Yang. A survey transfer learning, IEEE Transactions on knowledge and data engineering, 22(10):1345-1333 2009. 3
Xinlei Pan, Yurong You, Ziyan Wang, and Cewu Lu. Virtual to real reinforcement learning for autonomous driving. arXiv preprint arXiv:110.. 089522, 2017.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightee regression: Simple and scalable off-policy reinforeement learning. arXiv preprint arXiv:1910.00117 2019. 7, 18
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Guojian Wang, Quinson Hon, Xuyang Chen, and Lin Zhao. Data fusion-enhanced decision transformer for stable cross-domain generalization. orXiv preprint arXiv:2511.099
2025. 8, 18
Ruhan Wang, Yu Yang, Zhishuai Liu, Dongruo Zhou, and Pan Xu. Return augmented decision transformer
for off-dynamics reinforcement learning. arXiv preprint arXiv:2410.255) 2024. 2
Xiaoyu Wen, Chenjia Bai, Kang Xu, Xudong Yu, Yang Zhang, Xuelong Li, and Zhen Wang. Contrastive representation for data filtering in cross-domain offline reinforcement learning, In Forty-first International Conference on Machine Learning, 2024. URL https: https://openrevieet httpiiiwwweeeeeeeeeee fow.mrddd 3,
Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross- domain policy adaptation via value-guided data filtering. Advances in Neural Information Processing Systems, 36, 2024. 3
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, pp. 24631 24645. PMLR, 2022. 3
Zhenghai Xue, Qingpeng Cai, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. State regularized policy optimization on data with dynamics shift. Advances in neural informalion processing systems, 36, 2024. 3
Yu Yang and Pan Xu. Pre-trained language models improve the few-shot prompt ability of decision transformer. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https:
/ //openrevvew.net/eeeeeee www.:neettoommmddk0000000
3
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-1441 14142, 2020. 7, 18
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. international conference on machine learning, pp. 27042 27029. PMLR, 2022.
Zifeng Zhuang, Dengyun Peng, Ziqi Zhang, Donglin Wang, et al. Reinformer: Max-return sequence modeling for offline rl Xive preprint arXiv:2405.08740 2024. 2, 4, 8, 18
15

---

Published in Transactions on Machine Learning Research (02/2026)
A Proof of Theorem 4.5
Lemma A.1 (Corollary of Brandfonbrener et al. (2022)), Under the assumptions in Assumption 4.1, there exists conditioning function such that
(²(T)) J (T) I c ( + H²
aa
Lemma A.2 (Lemma of Brandfonbrener et al. (2022)). For any two policies n, n', we have || d S 2H Exngy [TV(T()))]+((())]
We define di N +N d3 + N N l. Define
Nº
No
L E sndggn P (-1a) DRL (P3 (-1s, g) [+|)s
Theorem A.3. Consider any function f : S + R such that the assumptions in Assumption 4.2 hold. Then for any estimated RCSL policy that conditions on f test time (denoted by &y), we have
J F - (*1) Or H² /L(A).
af
Proof. By definition and Lemma A.2, we have
J²((T) (+t) = (Rpr, [r(s. - Epy [r(s,a)))
s H ||a, day 11
$ 2 E Bonat [IV((((
Next, have
2 ERT
[V
1/ (alss - f(als.
Egndy
(f(ss))s Bondh (P())()) / (als, f()) - falss
$ af
E oo
f()) (()
Cj N NT
2 aj N /1 + +NT
E Bondmis pp (110) (Ttp:///.T/////./.sl:///aaaaaa..
C₁ NS + NT
of NS hI NT E grdiii Basddii..ggreee (11s) (0_ (4l)
Cj NS + NN
of NS ////1 NT 2l)
Proof of Theorem 4.5. Following the same argument in the proof of Corollary 3 in Brandfonbrener et al. (2022), have
( - J")) 0 C3 No + NT o
aj NS N9 NT 122 (ve(
1/4 NS NT + V Capprox )
log /0
Invoking Lemma A.1, we have
J(m) aj N8/1 + NT H² (Ve
N + NT
log |T|/8
+ V Capproxx + {H²)
NT + -NS
af
This completes the proof.
16

---

Published in Transactions on Machine Learning Research (02/2026)
B Detailed Experiment Setting
B.1 Environment and Dataset
In this section, we provide details of the environments and datasets used in our experiments. We evaluate our approaches the Hopper, Walker2D, and HalfCheetah environments, using the corresponding environments from Gym as our target environments.
B.1.1 Target Environment Dataset Creation
For the target datascts, we construct two distinct datasets: one containing a smaller amount of data (1T) and another with a larger amount (10T). The 10T dataset consists of ten times the number of trajectories as the 1T dataset.
Both Liu et al. (2022) and our work aim to demonstrate the following two key points:
The 10T dalaset represents high-quality data, whereas the IT dataset represents lower-qualily dala due to its smaller size.
Off-dynamics RL algorithms can enhance performance on 1T by effectively leveraging 10S source domain data through appropriate data augmentation.
Liu et al. (2022) creates the IT dataset by splitting the original target dataset (10T) based on timesteps, selecting the last 1/10 timesteps as IT. However, this approach introduces unintended bias in the Medium Replay setting, where offline trajectories are collected from a replay buffer in which the behavior policy improves over time. Consequently, the final 1/10 timesteps tend to exhibit a higher average return than the overall 10T dataset, undermining the intended quality distinction between 1T and 10T.
To address this issue and ensure fair evaluation of off-dynamics RL algorithms, propose uniform sampling method across trajectories in the target dataset. This approach ensures that the sampled IT datasot is a representative subset of the target data, free from biascs introduced by timestep-based selection. Notably, our method produces a 1T dataset lower quality than that of Liu el al. (2022) in medium replay setting. If an off-dynamics RT. algorithm can significantly improve performance on our 1T dataset. and achieve results comparable to the original 10T dataset, it would serve a more rigorous evaluation and a stronger indicator of the algorithm's effectiveness.
B.1.2 Source Environment Dataset Creation
We employ BodyMass shift, JointNoise shift to construct the source environments. The following descriptions provide detailed insights into the process of creating these source environments.
Mody Shift: The body mass of the agents modified by adjusting the mass parameters in the Gym environment. Detailed body mass settings are provided in Table 3.
JointNoise Shift: Noise is introduced the agents' joints by adding perturbations to the ac- tions during source data collection. Specifically, the noise is sampled uniformly from the range [-0.05, +0.05] and applied to the actions when generating the source offline dataset. Detailed joint noise settings are provided in Table 3.
For the source datasets, we utilize the BodyMass Shift and Joint.Noise Shift datasets from (Liu et al., 2022). Additionally, in our ablation study, we explore variations of BodyMass and JointNoise shifts be- yond those specified in Table 3. We also collect medium-level source datasets for the Hopper, Walker2D, and HalfChectah environments. Bchavior policies are generated by training agents with SAC using rlkit with checkpoints used for dataset collection. We construct the Ran- dom, Medium, Medium Replay, and Medium Expert datasets, each reflecting different performance levels determined by their corresponding SAC checkpoints. For the JointNoiss Shift setting, instcad of training a new SAC policy and collecting data through environment interaction, we introduce random noise within a specified range directly to the actions.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Body Shift and JointNoisc Shift in Hopper, Walker2D and HalfChoctah.
Hopper
Walker2D
HalfCheetah
Body Mass
JointNoise
Body Mass
Noise
BodyMass
Joint.Noise
Source
ma8s action +noise
mass|-1]=1.be mass| 1]-4
action|-l]+noiss action[-1]44
mass [1]5
action|-1]+non
Target
mass 1=055
cation[-1++)
mass [4]=1.0
action 1|
B.2 Baselines
In our experiments, consider BEAR (Kumar et al., 2019), AWR (Peng et al., 2019), BCQ (Fujimolo et al., 2019), CQL (Kumar et al., 2020), and MOPO (Yu et al., 2020), together with their DARA-augmented variants (Liu et al., 2022), as representative baseline methods. We additionally include recent approaches specifically designed for the off-dynamics setting, including DFDT (Wang et al., 2025), H2O (Niu et al., 2022), and TGDF (Wen et al., 2024). These baselines are compared against DT (Chen et al., 2021), Reinformer (Zhuang et al., 2024), and QT (Hu al., 2024), as well our proposed REAG methods.
B.3 Hyperparameters
In this section, outline the hyperparameters used for our REAG methodologies. The REAG approaches begin with dataset augmentation using either the DARA algorithm (REAG or the Direct Matching of Return Distributions technique (REAGMy) The augmented datasct is then used to train the DT-type frameworks, which is subsequently evaluated in the target environment. Specifically, for REAG dalaset augmentation follows the DARA algorithm, with its corresponding hyperparameters provided in Table 4. For REAGMy the augmentation process is described in Scction 5.3, where well-trained Conservative Q- Learning (CQL) model estimates state values, incorporating a clipping mechanism to mitigate extreme values. The hyperparameters for CQL training are provided in Table the elipping ratios listed in Table and the training parameters for DT, Reinformer, and QT adhere to the settings from their respective original papers.
Table 4: Hyperparameters used in Table 5: Hyperparameters used the DARA algorithm.
Table 6: Hyperparameters for the Clipping Technique Employed the REAGEv Algorithm.
in the CQL algorithm.
Hyperparamettr
Value
Hyperparameter
Value
SA Discriminator MLP Layers
Actor MLP Layers
3
Clipping
SAS Discrimiuator MLP Layers
Nonlincarity Function
Hidden
ReLU
256
4
Critic MLP Layers
Hidden Dimension
256
3
Walker2D Medium Replay
WalkerzD Random
Walker2D Medium
0.9 0 1.25
0.9
0.9 c 1.25
Optimizer
RMS
Nonlinearity Function
ReLU
Walker2D Medium Expert
0.9 < 1.25
Batch Size
256
Optimizer
Adam
Hopper Random
Learuing Rate
3 10-4
Batch size
256
Hopper Medium
0.9 < <
Ar Cocfficient
0.1
Discount Factor
0.99
Hopper Medium Replay
Hopper Medium Expert
0.9 4:
Temperature
1.0
HaifCheetah Randou
0.67 1.67-2.00 -2
Actor Learning rate
1 10-4
HallCheetahh Medium
0.67 <
Critic Learning rate
3 10
HalfCheetal
Replay
HaliCheetah Medium Expert
1.67.209.
B.4 Implementation Details and Computational Cost
In all experiments, the uncertainty parameter a used to determine the clipping bounds is estimated using a fixed budget of 1000 rollouts. This rollout budget is held constant across environments, datasets, and shift settings to ensure consistency.
We also report the wall-clock training cost of CQL to quantify the additional offline computation introduced by our approach. All CQL models are trained under a uniform hardware configuration with an NVIDIA RTX A5000 GPU (24 GB memory) and an AMD Ryzen Threadripper 3960X CPU (24 cores). Training CQL model on the target dataset requires approximately 2.48 GPU-hours, while training on the source dataset requires approximately 9.39 GPUJ-hous.
18

---

Published in Transactions on Machine Learning Research (02/2026)
C
Experiments
C.1 Additional Experiments Results
This scction presents more comprehensive experimental results, including additional variance information.
In Table 2, we provide the performance evaluation of our two return augmentation methods with the standard deviation. In Table 7, we present the experiment results with the 95% confidence interval (CI). The reported CI quantifies the uncertainty of the mcan performance and allows asscssment of the reliability of the observed improvements. We observe that, in most settings, our proposed methods improve over the corresponding DTpttype baselines and the improvement exceeds the CI radius. In some cases, the confidence intervals show limited overlap with those of the basclines, particularly under the Ropluy and Medium-Export settings on Walker2D, Hopper, and HalfCheetah.
In Table 1, we present the partial performance of various algorithms and their DARA variants in the Walker2D medium environment under BodyMass and JointNoise shift settings, considering both limited and sufficient target data scenarios. The complete experimental results are provided in Table 8. Additionally, Table 9 and Table 10 present a comprehensive comparison of different algorithms and their corresponding augmented variants in addressing the off-dynamics problem across various environments and shift settings.
DT REACHV
Reinformer REAG
QT
IT108
REACHT
1T108
REAG Raid
IT108
REACH
REAGE
BM JN
78 7685
80.83722.191
78.27733.08.
80.85-006622
82.35
80.566
84.325401.255
84.58250.629
S5.66&&1066,
M
71.08811200
27
71.79221188
74.4842.137
75,08411228 2211
74 1666
80.21411419
S080E11
78.67242.722 732.
W2D M-R
BM
73:06142.384
73.708#13437
67.365:099921
07.02277180
10.26617..602.
63.58845.32L
87. 87.29240.789
17.211115227
76
JN
38.253333944
33.72243.2941
62:22:004755
54.01=3.994
47
17.591
35.42946.0001
82.139+11277
82 2221
79.79645.845.
81:40011.022
88 88.236.2..311
83.52811.074f
83.088=1.001
11897
83.76140.912
93.0824114922
92.74
7777
M-E
115.746+1.80 385
111.000=2.790. 7901
111.2961.1155
117.300+0116
118.218418117
117.66533.102
116.14962.006
118.56440.8651
115.115423345_
BM 31.057.0:220
5335
37.787=223766
51.35744.610
59.08543.0651
51.77146.607
49.518412.164
51.796112.9791
6.92626666991
70330445540 5400
78335431331
7033025.752
72.36647.2961
70.4601-466287
08, 08,066488788 788
73.987110.881
08.709116.0000
Hp
M-R
BM
1.677
06 06.09233.2891
00.80
17.53118.329
20.952+12.150 1591
27.238115.810f
63 63.061117.3116
9.6667
#2.786:14.888
JN 61 0188000.300
77 1
43.9894663000
52.092122 468
93,706499388
03.40045 830.
51. 16.456:11,111 106.
M-E BM 33.531=1.000
33.63111.995f
08.973-9.326
14.266114.985
73.36349.5271
61.16244-6677
73.952440.2229
77.279 1001
JN
106:254:1100
965 100 3:6
10833133443
100,47240122 1281
100 109.256:002200 2304
109.09640.000 266
109
100.003007007
(09.746:40000 9691
M 39:95420.323
40 15125001.1111
37.50000.990.
38.26111.5371 12.65110.6101 37.35000
44.656411779
06.35311 47.30331:3955
440
44 44.144445500
500
07883:003631
482/4400331
43.00010.3811
48 20000
66.21340-06
52.394+1 7644
56
0..02610.009
He
M-R BM 20.96=11.927
27.81244.427
21020128191
31,58L11.529
25.99545.129. 32.1144118011
41.3004119777
42, 12.
1.335911.233
36.4085488
3841755000
38001:4:3811
4022063.618
41840+3.670
1924
53 03,16340.944
53.80011218
218t
73.267
M-E BM 51991#1.21 36.22333.537
51.3571102188
10.508
35.81822.2981 06.188220577
71.080110.927
70.53319.959 69.81916.356..
JN 101573-10:875 20200-22001
11:51+333641
76.073448144
79 70.3000.1800
78 188081114871
92.96114449988
83.602+0.088
$2,148+3,2244
Table 7: Performance evaluation of two return augmentation methods, REAGMV and REAG Dara' integrated with DT, Reinformer, and QT frameworks in off-dynamics scenarios with 95% Confidence Interval (CI). The experiments are conducted in the Walker2D (W2D), Hopper (Hp), and HalfCheetah (He) environments under the Medium (M), Medium-Replae (M-R), and Iedium-Expert (M-E) settings. The source environment is modified using two shift conditions: Mass shift (BM) and Noist shift (JN). For reference, the table also includes the performance of the original DT-type mothods without augmentation, displayed in gray font. Performance changes due to augmentation are indicated with red upward arrows (1) for improvements and green downward arrows (1) for degradations compared to the original DT-type methods. All reported values are averaged over five random seeds (0, 1012, 2024, 3036, 4048).
C.2
Ablation Study
C.2.1 Complex Dynamics Shifts
To evaluate the behavior of the proposed method under more complex dynamics shifts, we conduct additional ablation studies on the HalfCheetah environment with the following modifications:
Kinematic Shift. The rotation range of the back thigh joint is modified from the default interval [-0.52, 1.05] to a substantially reduced range [-0.0052, 0.0105]. This change constrains the feasible joint motion and alters the transition dynamics by limiting the agent's 's actuation capability.
19

---

Published in Transactions on Machine Learning Research (02/2026)
BEAR
AWR
BCQ
CQL
M
M-R
M-E
M
M-R
M-F
M
M-R
M-E
M
M-R
M-E
W2D
0.777
68 aaar
100206iesm
00,638.000
101660aaze
57 4720m
101:611a01
10T
13.1
6,852.
21.
78.000
18
109.154
61.735,a50
18.99
10
HI
1T
8.770gnaa
2.2
56,209
51.230g1.2
51.00661.00
68.
118
11.101a2m0s
82, 12u07
101'
20.30
68. M0
64 2417
57 1i
105 361 21.32
73 2a
60
270
82 20000
102 aame
1T
3.089.a 105
90.168.1.xm
41.05L
60.173--11
26.965 a0
51 aa
He
10T
10.837amm
13.
42. 12.30amm
41.041
90.2/2aaaee
46. 18
08.575u220
58.535
49. 18.382
48.900anom
u7.650umm
MOPO
DYT
Reinformer
QT
M
M-R
M-B
M
M-R
M-E
M
M-R
M-E
M
M-R
M-K
20.363
20.310.g es
07 a7
31.082...00
107.171..on
78.034. aw 38.072..m
100.284.... on
817788.1m
67.588 111.722,
W2D
10T
22.201apmm
18.52
19,
08.525
108 1886622a2a
81.777a100
08.198
109.845
88, 28.2022
8500224mm
111.2M4anos
1T
31
6.849
35, 202
66 07.
61 3..
100.71441m
74.757
36 36,008amm
60
83. Mlligams
108 25000
HIp
10T
32
760
36.161e2a0 8.
86 50.5800ssa
700
108006
77. 792
39
79
79,380424m
100821ai 10
in
1T
12.277 24.3
25, 0.055 ass
41. a0:0
15.154
77.500
42 42.938
18,400_1.
72.085ascen
32318.2445 50.4444017
87.854
He
10T
65.863.13
28 28.221.6
42 47.
34 50588
82.844 74.8
40, 233333
30.434.aae
87. 87.378.030
51.28410/05
40.087a3304
96.116
Table 8: Performance comparison of algorithms on the IT. 10T, and 1T10S datasets. In this study, IT)0S(B) refers to the source dataset under the BodyMass shift setting, while (J) corresponds to the source dataset under the JointNoise shift setting. The experiments are conducted in the Walker2D (W2D), Hopper (Hp), and HalfCheetah (Hc) using the Medium (M), Medium Replay (M-R), and Medium Expert (M-E) datasets. All reported values are averaged over five seeds (0, 1012, 2024, 3036, 4048).
BEAR
AWR
BCQ
CQL
MOPO
D-BEAR
D-AWR
D-BCQ
D-OQL
D-MOPO
DFDT
IGDF
BM 5.76 H: 1.000 17.48 e 0:30
70.01
5:000
7A317 4 1.398
21017 6 1.97
6506 + 3220
T6OH 4:00
71105 + 0.955
7L156 H 2501
B1001 = (00)
MI ML353 0.001
61101 4 6.858
76.771 + 2.01
Walker2D M
JN
4500 1: 1418 00,556 a 1,065
63006
L167
08369 4 0.65
23.02 à L148
0.68 à 1.54
54.00 a 0.01 01081 4 1.15
60.14 A 0.00
25,000 = 0.66
10.333
5/000
12540
13,022 90.40 4 2A11
Walleraa M-R
BM 000 à: 1.66 40.055 a 2,20
1998
56755 à 0.335
11:00 à 2.61
1008 2053
52008 à 1.20
51.41
5,05
574422
0.000
12.121
= 2.06
10.05
15.10 2: 9.2 02.144 2 4081
0.474 4:070 R0.005
8.501
50.011
1.00 50.00
11,07
0.500
0.284
36.177
2.42
00.74
50.522
15,39
T1.336
50.220
06334 7,75
S116 1:0001
108
Waller2D M-E
145 # 1,33 154.022 2.00
112
= 1.49L 304010 0.5%
17.02) + 0.020
14,303 = LoO 900 1.55 111:201 1100
1.305
10:033
108,000 d 0L707 05,802 # 0.77
111
1.200
BM 22 436 # 0.033 25,545 = 0.225 24853
LO0 20054 d 2,20
20,763 # 3,500
20,005 = LOGG 20.234 # 1207 20.457 # 1.365
40.001 8 0.942 21.455 = 0.868
06L333 =: 3,840 100.704 = 2.207
T6.226
: 5.112
Hopper M
JN 6,006 1 1435 07.001 a 0933 71509
0.00 71 196 4 0/1
23,300 1 1.599
10:50 1. 1.02 53.933 5 8.0 74 0.555
65.011 1 136 21:000
- 0.94
50.703
7.181 101 311 0.007
75-157
BM
6.22 à 0.102 5GL507 a 2,30
64.20
0.515
01455 a: 0:93
1.504 = 1,701
2.00 A 1.328 44,883 # 1220 G1116
0.251
65.153 a: 0.000
0,482 d: 1.061
74720 11.422 07 07.888 d: 2.473
56:855
= 2.70
Hopper
JN 1:811 3851
1/005 65
1.317
5,405
0.508
5.65T 11251 15.977 3850 G151D
1.000
65.175 6: 1.918
0.317
74
11 958888 4 1.415 10.330
3.508
Hopper M-B
5021 57.100
0611
10.511
36666
30.003
0.003
0.000
110014
70/00-
10000 572 3.901
20.61 à 1879 74.706 a 1.80 100.500 20028
12.022
0.081
0.812
S8.022 is 0.36 500912 de 0.800
111.80
04.28
1.20 50.001
1005
04.00
à 18.254 100.657 2810 91.67 1: 23.644
HaldChwwtah M
BM 5.511 +: 1.518 (2.360 e 0,902 35.035
0.407
07.053
d: 0.358
58,897 #: 1.440
0,000
1.705 41,000 d: 0.000
23.000 d: 0.555
37.189
0:0118
00.111
= 0.00
(0,559
e 5.557 51.00 #: 1.508 47:335 e 0.331
JN 1,58 è 1.06 41,590 e 0,702 30,011
0:37
20046 d: 0.031
53.073 de 0,310
2.00 de 1.822 42,045 de 0,701
02.10
di 0.457 43/354 à: 0:000 E3AATT =: 0.734
40.760
0.245 5T.00 #: 1.45
06647 e 0,20
HallChwwtsh M-I
BM 7,425 1. 1.500 15.66 5,333
32,505
1. 266
50.506 1 0.500
50.02)
1. 1.506
43500 11552
17-118
L. 3.000
32,006
1.336
51/23
1.0:000
0.021
56.525
57.835 1. 1.866
06:355 0:220
18.30
84488 RL.742
41193
42,007
2508
G1.06
0:266
$1,008
L.544
17,02)
58:105
1.75 20.00
0.545
52001
50,153
57.776
46812 0.157
75
08:13 0.000
HollChootta M-E
3106.1 1 0.39 18 2.063
ENB 10095 8 600 4. 1.108 501000 10101
750
(1777 19/7N) à 800 71 DII 4.301
REPIO L. 0.066 10014 7011
60 3.001 01.000 4 1.10 08197 0.353
Table 9:
Performance comparison of conventional offiine reinforcement learning methods BEAR, AWR, BCQ, CQL, and MOPO-an and their DARA-augmented variants, together with recent dynamics-aware baselines (DFDT, H2O, and ICDF), under Mody and Noint distribution shifts on the Walker2D, Hopper, and HalfCheetah environments. All methods are evaluated on the Medium (M), Replay (M-R), and Medium-Exxper (M-E) splits of the 1T10S dataset, which consists one target dataset (1T) and ten source datascts (10S). D-XX denotes the RARA-augmented variant of algorithm XX. DFDT, H20, and IGDF are included as representative off-dynamics basclines and are evaluated under identical experimental settings.
These settings introduce structured changes to the underlying system dynamics beyond those considered in the main experiments. The corresponding results are reported in Table 11. Under the kinematic shift, all methods achieve comparable performance across DT, Reinformer, and QT, indicating that restricting joint motion primarily limits action expressiveness without fundamentally destabilizing the learned policies. In this setting, the proposed methods consistently match or improve upon their corresponding basclines, with REA QETAQQ yielding the strngsttt performance.
20

---

Published in Transactions on Machine Learning Research (02/2026)
DT
Reinformer
QT
REAG
REAC
REACHY
REAGEE.
REAG
REAG
QT
Darn
BM 78, 708 +: 1,233
80,857 1: 0.509 845255 4: 0,425
80,857
1.715
823354 d: 1,479
81.582
0.507
78.257 +: 2:425
80666
0.506
83.08
0.800
Walker2D
JN 71.088 + 1,02
74.748 + 1.71
80691 + 1,143
76.008
1894
75008 + 0.986
80.004
1,00
71,779
1.706
74.968
1.941
78.672 2.01
BM 73,661 11 1,920
67,002
1. 3,707
87:292 1 0.031
73,708
1,570
50,256
1. 14.211
87.491
1.226
67,565
0.799
06.658
1,308
76.169
L. 7.367
WalkerDD
JN 38,255 +: 3.181
36,901 4: 3.277
82.089 4: 1,000
56,722 = 2,633
47,301
+: 10.244
82.303
4.206
62,226
53-438
4.833
70,706
708
84.490 0,893
83.98 + 0.806 93007 + 0.948
88 /3/2355--1
1,886
84.807
1.117
02:744-
0.400
85.39
0.805
83.761
0.735
04.578
1.389
WalkerDD M-E
JN 115.746 1.116
117,560
2,550 116.1/9
111.000
2.207
118:215
118.561
0.607
111.206
117:705
2/659
116.115
BM 34,057 +:
51:957 +: 3.733
49.516
9.708
30,635
1,230
50:065
2,700
51,706
9.977
97.767
1.014
51.
6.322
5.348
Hopper M
JN 70,685 à: 0.728
70,310 4. 4.63 68.659 4: 7,079
70.350
3.657
72.310
5.877
73,987
8,080
78,525
4.442555 2.522
70.106
3.722
68.709
12 100
BM 61,216 1: 1,501
17,581 L. 6.735 160
13,918
66,092
0.233
20.962
9.791
76.257
7.8L0
60.308
1,086
298
1. 12.755
82,796
Hopper M-R
JN 61870 = 0.240
41.820
18773
93,204 a 7,500
77.825
1.008
43.965
5,075
90,400
4,006
63,326
1.728
02.0022 a: 10:005
51.450
12 108
BM 33,551 1: 0.846
68,973
A: 7,512
61.102 a. 3.767
52,873
0.151
01,200
1: 12,070
73,952
a. 10.294
33,631 A: 1.005
73:303 a: 7.074
77.279
18
Hopper M-E
JN 086,251-2 1,583
109:256
0.125
1090056
0,214
109.967
1.081
109 472 L. 0,100
100,800
800.803-.
_. 0.609
108 261 1. 2 012 109.255
1. 0.188
109.746
0.771
BM 39.951 # 0:200
31,930 #: 0.483
44.086 4: 0.943
40,230
= 0,911
42.451 da 0.491
47.003
0.318
37,009
0.336
38.261
4: 1,238
40.363 = 0.358
HaRCheetah M
JN 17.725 1: 0.431
18,271
1. 0.191
592233 4. 0.327
11.119
3,672
13.009 1. 0.307
32.391
1.113
17,833
0.281
68.101
0.168
36.120
A. 0.410
BM 20,996 1. 9:007
$1,581 1. 1,248
(1,300 1. 0,787
27.812
3.256
32.114
1/455
42 00.0055.0
0.729
24,050
2:271
26.005
1,378
41.339
0.985
HalCChoetah M-R
36,000
40.290
53,233 a 0.750
38,417
40.840
2,800
33,870
0.981
3.929
38.
3.577
50.257
0.386
51981
40.568
71.088
8,802
56,228
16.018
L,657
5.12n
6.231
70,553
8.022
HalfCheetah M-E
JN 70.733 8.209 76.077 3818 82961 4.319
77.702
2.099 79:300
0.140
83,632
0.009
17.751 2.702
18.081
1.198
82,148 2758
Table 10: Performance comparison of traditional offline reinforcement learning algorithms, including DT, Reinformer and QT, along with our proposed methods REAGET and Noise distribution shifts in the Walker2D, Hopper, and HalfCheetah environments. Evaluations are conducted using the Medium (M), Medium Replay (M-R), and Medium Expert (M-E) settings of the 1T10S dataset. The 1T10S dataset comprises a 1T (target) dataset, and a 10S (source) dataset.
MV: REAGE Dara' REAG MV REAG REAG MM and REAG Dara under BodyMass
Dars
Method
Baseline
REAGMV
REAG Dars
DT
42.277 + 1.062
42.752 + 2.387
41.257 + 4.437
Reinformer
42.929 + 0.539
42.826 + 0.558
43.092 + 0.442
QT
43.968 + 2.037
47,698 0.211
43.896 + 3.677
Table 11: Performance comparison of traditional offline reinforcement learning algorithms, including DT,, Reinformer and QT, along with our proposed methods REAGET REAGET Dara REAGBBea REAGReitt REAG My and REAG Dara. under Kinematic shifts in HalfCheetah euvironment. Evaluations are conducted using the Medium (M) setting of the 1T10S dataset. The 1T10S dataset comprises a 1T (target) dataset and a 10S (source) dataset.
Dara
C.2.2 Sensitivity to Q-Function Quality
REAG learned Q-functions for return relabeling. To evaluate its sensitivity to Q-function quality, we vary two factors in CQL. training: (i) the conservative regularization coefficient and (ii) the number of target samples used to train the critic.
All experiments are conducted on Walker2D-Medium with BodyMass dynamics shift. For each setting, CQL critics trained under different a values and dataset sizes are used for return relabeling in REAG while all other components are kept fixed.
As shown in Figure 7, REAGMya similar performance across broad range of a values and training sample sizes. Although CQL training becomes less stable under strong regularization or limited data, the final policy performance varies only marginally. This suggests that REAGhyis not highly sensitive to the quality of the learned Q-functions.
C.3 Gaussian Approximation of Return Distributions
Our approach models trajectory-level returns using a Gaussian distribution. This assumption is supported by empirical analysis on standard offline RL benchmarks.
21

---

Published in Transactions on Machine Learning Research (02/2020)
0.2, 1000
a= a=0.02,5- 1000
a=0.2, 100
a 0.02, 100
Performance
85
80
75
70
RADT
RADT
RADT
Figure 7: Performance of REAG under different CQL training settings on the Walker2D-Medium environment with BodyMass dynamics shift.
hopper-medium--2
halfcheetah- wew.eettaa--mii----2
Trajectory Cumulative Reward Distribution
Trajectory Cumulative Reward Distribution
Count
500
1000
1500
2000
2500
3000
0
1000
2000
2000
4000
5000
Cumullaive Reward
Curuulative Reward
Figure 8: Empirical return distributions of trajectory-level cumulative rewards in the D4RL medium datasets for Hopper and HallChoctah.
We examine the D4RL medium datasets, which are collected using medium -erttuum--effommance SAC policies. For the Hopper and HalfCheetah environments, we compute the cunulative reward of each trajectory and analyze the empirical return distributions. As shown in Figure the resulting histograms are unimodal and approximately symmetric, without pronounced skewness or heavy tails.
These characteristics indicate that Gaussian distribution provides a reasonable approximation of the ob- served return distributions in these datasets. While this assumption not expected to hold in settings, is appropriate for the benchmarks considered in this work.
D Analysis of Failure Cases
REAG improves target-domain policy learning by augmenting training with relabeled source trajectories. Its effectiveness depends on the degree of alignment between the source and target domains, particularly with respect to state-ace action occupancy. This dependency formalized in our theoretical analysis through the domain occupancy overlap coefficient. (Assumption 4.2 and Remark 4.6), which characterizes the conditions under which source data can provide reliable information for the target task.
When the source target occupancy overlap is limited, return relabeling may introduce bias. In this regime, relabeled source trajectories can distort the estimation of return statistics and value functions, resulting in de- graded downstream performance. This behavior observed for Reinformer on the Walker2D Mediun-Replay dataset under both BasyMass and JoineNoise dynamics shifts, where REAG shows limited effectiveness and may negatively affect performance.
A contributing factor is the structure of Medium-Roon Rediuum-Replay datasets, which are generated from mixtures of behavior policies and exhibit substantial heterogeneity and multi-modality in both state visitation and
22

---

Published in Transactions on Machine Learning Research (02/2026)
return distributions. In such settings, the target subset (1T) is often insufficient to reliably characterize the target return distribution, which can deviate from Gaussian form in parts of the data. Consequently, the return statistics and Q-function estimates used for relabeling become noisy, reducing the reliability of return augmentation.
Based on these observations, REAG best suited for settings with sufficient mource-targed occupancy over- lap and relatively stable return distributions, such Medium and Medium-Expert datasets. For highly heterogencous replay datascts, aggregation-basee variants such REAGDars or disabling return relabeling may be more appropriate.
E
Additional Theoretical Results
Recall that in Section 4.3, we assume a single behavior policy to collect offline dataset under both source and target environments. This simple setting is studied to provide principled motivation and conceptual guidance for the proposed method. While in practical implementations, we use different policies to collect the source and target datasets, leading to gap between our theoretical analysis and practical implementation. In this section, generalize our thcoretical analysis and results to the case where the source dataset and target dataset are collected by different behavior policies, thus aligning with our empirical impletentattin.
We generate the source dataset D by 3s and target dataset D by Br, and denote their sample sizes as N and NT. Denote the occupancy mcasure over state induced by Bs under ps as dág
ds and by Br under pd
No dT + as d Define the mixture occupancy measure as d == NS+NT Br No
No N de and the mixture policy
as 3 = N= Br + N Ss. For simplicity, we denote c c No+NV is and thus B = cßs + (1 c)
N
Recall the definition ( (g]s, a) is the distribution return induced by Br under pT, We define the goal policy of interest as
(o=
[a(als) ENEA (0 [(s))s
We define PT
(())), d()(d)
We make the following assumptions, which corresponding to Assumption 4.1, Assumption 4.2 and Assump- tion 4.3 in the main text.
Assumption E.1. (1) (Near determinism) (r # r(s, a) or + ((s,a))s,a) S c at all 8, a for some functions and r. (2) (Consistency of f) f(s) = f(s')++++)))) +r for all s.
Assumption E.2. For all we assume (1) (Bounded occupancy mismatch) pf (x) S C2 C (8): (2) (Return
coverage) (Return coverage) Ja (s(als)PTT (+|s, d) 2 ay and J. 3r (a (als) (1)s 2 ay; and (3) (Domain occupancy overlap) d' (s) S 7yds (. (s).
(als, log (aa s' s',g')ll S for any ", 8. and all E II. (3) The approximation error bounded by capprox, i.e., minren L(T) s Capprox.
Assumption E.3. (1) The policy class II is finite. (a, (,,8,g,a',
is the policy we want to learn. In the following theorem, we show that this policy is near optimal under the target environment.
Thcorem E.4. Under the assumptions in Assumption 4.1, there exists conditioning function such that
1 + +3) H..
--(r-- S
Proof. The general proof idea follows the proof of Corollary 1 of Brandfonbrener al. (2022). But since our policy Ff is not classical RCST. policy, need careful analysis on the derivation.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Let g(s1, 01:H) be the value of return by rolling out the open loop sequence of actions d1F under the deterministic dynamics induced by pf and r. Then we have
E,,15(81)] J(rr) Es, [ (+1) - 91]]
= = [B /s [x1) - g(s1, 01.0): + Es 10(s), a1.H) - gill
Bs, (Ba (f(s))) g(s1, (1.1 ell²
(E.I)
where the last step follows by bounding the magnitude of the difference between 91 and g(s1, (1:H) by and applying a union bound over the H steps in the trajectory. Now we consider the first term. Again bounding the magnitude of the difference by H get that
/ P (ann)s)) (g((), # H(s)))
ss E Bes (ee ((11, S Bsy Jain
(F.2)
To bound this term, will more carefully consider what happens under the distribution PT To simplify notion, let & = (s1, d1æ-1) be the result of following the deterministic dynamics defined by 5 up until step Expanding it out, applying the near determinism, the consistency of f, the coverage assumption, we have
PH = ( / pa (12131,, a1) (12.5131,882)
S (T((aa11) (02 52) +e
PT (9) f(s ((11)(s,,a))
= B((1is)))
Pay Pa (22.1181,32) +e
Σ En'es Br (g1 f(s)))
6 P. B7
(0) - [(1) ((1
=((a)181)
Pa (a2:1101, 32) +
Evec PT (g) f())((()))))))) [()))s
E + PT (g2 f(32) 52)
=1(
Pay (22.H|51, 82) te
B,,, (9) = =[(81)
PT (g2 32)
< <((11181)
Pr,y (02:11111 11, + <(1/a/ +1)
PJ B.8T (91 {(81))81)
PTr (g2 (522) P3 f( (33)
52 pd Pp (6) [(8))(81) pd 8. (g2 f(52) 82) Pr, (a3:H/51, + 26(1/af +1)
PT (g2 = =f (32))
< B(arls1)8(0215)) 3(ax|5H
p.gr (91 = (811){81)
PT (93 = (33))
Pf (gn
Pf (9H-1 = f(3H-1)(s=-) He(l/ay + 1).
Pb,g (92 = ((22)
By the fact that
Pl.er (gn = [(3)))5m)
Σ
PT (gh (gh = f(()))()))))))))))) d') ((a)(5n))
W'EA
Σ
pd (gh = [(Bn)) 5r, d') (c: 3s (s(((())) + (1- (a((5)))
d'EA
2
Σ
PT (gh
= -(8n)) a') (1 - c) 3r (a'((5))
w'E
(1 c)
By (9h- = ((x))
24

---

Published in Transactions on Machine Learning Research (02/2026)
have
PT (9n = (() 1
Ps my
(01:H(1))
2 (())55))
P.,, (91 = f(si) 81) (1 c) HH((//ay+ +
Br
1=1
1 [(g(s), a1.1) f(si)} I (()l)))
H
Pe (0l f(s1)[s1) (1 (1- + He(L/as + 1),
where the last step follows from the determinism of the trajectory that determines 8yr and the consistency of f. Plugging this back into (E.2) and noticing that the two indicator can never both be 1, have
E,, 1o (f1) - g( (1.H)] - H² (1/aa++1))
Plugging this back into (E.1) finishes the proof.
Now we are ready to state and prove our main theorem.
Thcorem E.5. There exists a conditioning function f such that Assumptions E.1 and E.2 hold. Further assume Assumption E.3 holds. Then for some E (0, 1), with probability least 6, we have
JT J=(') $ 1 Gj N NT
log | 1/4 N /1 NT H² (Vel
NT + NS /Cappeox)
' H³). (E.3)
as
Define
L(a) = E gradmi ga..ddn...ppppp.p. g p, (-1a) DKL
To prove Theorem E.5, we first prove the following result.
Theorem E.6. Consider any function : such that the assumptions Assumption E.2 hold. Then for any estimated ROSL policy that conditions on at test time (denoted by fy), we have
CmT 22L))) af
Proof. By definition and Lemma A. we have
Epz (²(()) - 5= (+t)) = H (Epr, [r(ss a)] E -Ep [r(s,
< H I|da, day ll1
< Esndt [
Next, have
2 E gn
(1s))
= E.wdy 1/1 [Tr(als) - -
PT (11s)
/. ( ,, (-1s)
BEndy
3,8r (:18)
C1
Eyndy Bn pp 3.0 (1s) (1))
as
C₁
N + NT
E Baniiii go pPP B.0y (14) [T((
0j N5/111 // + NT
Cj N +NT
0f NS //g + NT -E Banndmia gn
Pa PT. (110) / (2K(())(()))
Cj N +NT
aj No //S + NT 22...)
25

---

Published in Transactions on Machine Learning Research (02/2026)
Proof of Theorem E.S. Following the same argument in the proof of Corollary 3 in Brandfonbrener et al. (2022), we have
((t)) (
Or N N of NS by + NT H² (o (Vc(
log M//6 1/4 NS + NT
+ /Capprox)) )
Invoking Theorem E.4, we have
N + N (())))=()))0(( aj N5171 NT 112 (vol
log |T| /
1/4
+ +-1
NT NS
+
Capprox
af
This completes the proof.
26


Reviews:
[{"review_id": "4DLcHhe2yO", "reviewer": "Reviewer_bpTG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies offline off-dynamics RL with abundant source-domain data and scarce target-domain data, extending return-conditioned supervised learning (RCSL), particularly Decision Transformer (DT), to leverage mismatched dynamics. It proposes return-augmented RCSL (REAG), with two instantiations: (i) REAG-DARA, which applies density-ratio–based per-step augmentation summed into returns, and (ii) REAG-MV, which aligns source and target conditional return distributions via moment matching using offline Q estimates. A finite-sample analysis shows that, under standard RCSL assumptions and domain overlap, incorporating augmented source data improves the bound over target-only learning.\n\nStrengths\n- Novel extension of off-dynamics augmentation to return-conditioned policies, where standard reward correction methods (DARC/DARA) do not directly apply.\n\n- Clear return-distribution alignment perspective for RCSL, with practical instantiations connecting prior trajectory matching and DT-style learning.\n\nLimitations:\n- The REAG-MV theory assumes access to conditional return CDFs under the same behavior policy; in practice, moments are approximated via learned Q-functions and across-action variability, which weakens the connection to the theoretical formulation.\n\n- The “same behavior policy” assumption is violated in the datasets used, so theoretical guarantees for the implemented estimator are unclear.\n\n- Dropping DT’s return-consistency constraint improves performance but changes return-token semantics and may risk brittleness or extrapolation issues.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Mostly yes. The empirical results are generally convincing and consistently show that the proposed return-augmentation methods improve DT-type baselines across multiple off-dynamics benchmarks and ablations. However, for REAG-MV in particular, the theoretical claims rely on strong assumptions (shared behavior policy, access to conditional return distributions) that are violated in practice, and the implemented estimator only loosely corresponds to the theory.", "audience_yn": "Yes", "audience_explanation": "Real data rarely comes from the exact environment you care about. The paper shows how you can reuse lots of existing data collected under different dynamics instead of throwing it away. Both the theory and experiments suggest when mixing source and target data is beneficial, which is useful for researchers working with limited target data.", "requested_changes": "Methodological issues\n- Missing comparisons to several strong and recent off-dynamics baselines tailored to large dynamics gaps: model-based/planner approaches (e.g., OCDP), target-dynamics modeling (MOBODY), robust cross-domain Q-learning (DROCO), and sequence-model cross-domain pipelines (DFDT). Including at least one of these would situate gains more convincingly.\n\n- Sensitivity to the quality of the learned Q used in REAG*MV is only lightly probed; CQL trained on tiny target sets can be unstable. More systematic sensitivity analyses (vary target size, critic seeds, α values) would strengthen claims.\n\nPresentation Issues\n- Notation occasionally shifts (g vs g(tau) vs g(s_t) Ppi vs ppi) and equations include minor typesetting issues that obscure exact meanings (4.6)\n\n- Some tables contain formatting artifacts (arrows embedded in text, inconsistent symbols) and are hard to parse; figures lack axis units in places and have inconsistent labeling (“REAGMY” vs “REAGMV”).\n\nMissing related work or comparisons\n- There appears to be substantial overlap with prior work on Return Augmented Decision Transformer (RADT), which proposes conceptually similar RADT‑DARA and RADT‑MV variants and a closely related theoretical result. This paper does not cite or differentiate from that work; clarifying novelty, differences in theory/algorithms/experiments, and empirical deltas is necessary.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W9HiHvzlNU", "reviewer": "Reviewer_tdvw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the dynamics shift problem in return-conditioned supervised learning (RCSL), highlighting challenges arising from the peculiar structure of the RCSL policy class and the lack of a straightforward policy representation. To mitigate these issues, the authors propose a return-augmented approach for Decision Transformers, in which the return signal in the source domain is modified to align its distribution with that of the target domain. They introduce two instantiations of this approach: REAG_Dara and REAG_MV. The authors further provide theoretical guarantees showing that, under this alignment, the learned policy achieves the same suboptimality bound as a policy trained directly on data from the target domain.\n\nNote: I am not familiar with this line of research and cannot asses the novelty or positioning of the paper in literature.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper provides theoretical and empirical evidence to support the claims.", "audience_yn": "Yes", "audience_explanation": "This paper introduces a method to improve performance of decision transformers in off dynamics reinforcement learning, which is an interested topic in the community.", "requested_changes": "Minor issues : There is a typo in preliminary section - plolicy", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jOlYKUyEnc", "reviewer": "Reviewer_7hkf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Return Augmented Decision Transformer (REAG) for offline off-dynamics RL, where a large source dataset with shifted dynamics supplements limited target data. Prior reward-augmentation methods (DARC, DARA) designed for dynamic programming–based RL cannot directly transfer to return-conditioned supervised learning (RCSL) methods like Decision Transformer, since RCSL policies explicitly condition on return-to-go. REAG augments returns rather than rewards by transforming source returns to align with target distribution.\n\nTwo implementations are proposed: REAG*_Dara adapts DARA's classifier approach (ψ(g) = g + η∑Δr), while REAG*_MV matches return distributions via Gaussian approximation. Theorem 4.5 shows suboptimality improves from O(1/√N_T) to O(1/√N_S) when N_S >> N_T. Experiments on D4RL MuJoCo (Walker2D, Hopper, HalfCheetah) with BodyMass/JointNoise shifts show both variants improve DT, Reinformer, and QT, with REAG*_MV generally outperforming REAG*_Dara.\n\n**Key Strengths:**\n- Clear motivation adapting augmentation techniques to RCSL policy classes\n- Principled derivations with finite-sample guarantees matching non-shifted regime\n- Comprehensive ablations on shift magnitude, clipping, consistency, value function quality\n\n**Key Weaknesses:**\n- Incremental novelty—REAG*_Dara directly translates DARA; Gaussian approximation lacks justification\n- Missing recent off-dynamics baselines (H2O, VGDF, PAR, BOSA); comparisons only to 2019-2020 methods\n- Inconsistent improvements (~30% degradation in Table 2) with no significance testing\n- Narrow scope (MuJoCo locomotion only, two simple shifts, ad-hoc clipping bounds)", "claims_and_evidence_yn": "No", "claims_explanation": "The theoretical claims are sound but straightforward applications of existing results. The Gaussian approximation (Eq 4.4) has no validation of when it's appropriate. The experiments have some gaps:\n\n**Missing state-of-the-art baselines**: Comparisons are against 2019-2020 methods (BEAR, AWR, BCQ, CQL, MOPO) augmented with DARA. Recent off-dynamics methods (H2O, VGDF, PAR, BOSA, IGDF, SRPO) mentioned in related work are absent. Without these, we can't tell if REAG is actually competitive with current approaches.\n\n**Inadequate statistical rigor**: Table 2 shows high variance with ~30% of results degrading (marked ↓). Example: Walker2D M-R JointNoise REAG_MV^Reinf gets 47.591±10.244 versus baseline 54.801±3.217. The paper claims \"consistent enhancement\" but the evidence shows otherwise. No significance tests despite variance often exceeding mean differences.\n\n**Limited scope**: Only MuJoCo locomotion with two simple shifts (mass scaling, action noise). The claims need validation beyond this narrow setting.\n\n**Implementation underspecified**: Clipping bounds (Table 6) look ad-hoc with no principled method. Number of rollouts for σ estimation not reported. No wall-clock time analysis for CQL training overhead.\n\nThe ablations (Section 5.4) are solid, and value function validation (Figure 6) is convincing. But missing modern baselines and lack of significance testing are serious problems.", "audience_yn": "Yes", "audience_explanation": "The paper tackles a real gap: applying Decision Transformer to off-dynamics RL. This matters for robotics sim-to-real transfer and any domain where target interaction is expensive. The insight that reward-augmentation techniques need adaptation for RCSL is useful and helps practitioners understand these methods. The finding that return-level augmentation can beat reward-level approaches for DT is worth knowing, as is showing these methods can compete with traditional offline RL (Figure 1).\n\nThat said, the contribution is incremental, in that it adapts existing ideas rather than new insights. The scope is narrow (just MuJoCo locomotion). That said, it's a concrete contribution that extends DT methods to an important setting.", "requested_changes": "**Major**\n\n1. **Add recent off-dynamics baselines**: Include 2-3 recent methods (H2O, VGDF, PAR, or BOSA) in main experiments to show how REAG compares to ~state-of-the-art.\n\n2. **Statistical significance testing**: Report p-values or confidence intervals for Table 2. \n\n3. **Fix overclaims**: Abstract and Section 5.3 claim \"consistent enhancement\" but ~30% of Table 2 shows degradation. \n\n**Misc**\n\n4. **Justify Gaussian approximation**: Show histogram plots of return distributions or provide theoretical analysis of when this approximation works.\n\n5. **Expand experimental scope**: Add one non-locomotion task and one complex dynamics shift (morphology, contact parameters) to show this generalizes.\n\n6. **Failure mode analysis**: Explain when REAG degrades performance (e.g., Reinformer in Walker2D M-R) so users know when to apply it.", "broader_impact_concerns": "No Broader Impact Statement needed. The main risk is that better sim-to-real transfer might lead to deploying insufficiently validated policies in safety-critical applications, but this applies to all RL methods. No major ethical concerns.", "additional_comments": "Overall, I think this is work is useful: extends DT to off-dynamics settings. Adding modern baselines and making experiments more thorough would make this a solid paper for TMLR.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)