MOBODY: Model-Based Off-Dynamics Offline Reinforcement Learning
Authors: Yihong Guo
Yu Yang
Pan Xu
Anqi Liu
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
off-dynamics RL
domain adaptation
model-based RL
Primary area: reinforcement learning
TL;DR: We propose a model-based method for off-dynamics offline reinforcement learning problem to explore the target dynamics.

Abstract: We study off-dynamics offline reinforcement learning, where the goal is to learn a policy from offline source and limited target datasets with mismatched dynamics. Existing methods either penalize the reward or discard source transitions occurring in parts of the transition space with high dynamics shift. As a result, they optimize the policy using data from low-shift regions, limiting exploration of high-reward states in the target domain that do not fall within these regions. Consequently, such methods often fail when the dynamics shift is significant or the optimal trajectories lie outside the low-shift regions. To overcome this limitation, we propose MOBODY, a Model-Based Off-Dynamics Offline RL algorithm that optimizes a policy using learned target dynamics transitions to explore the target domain, rather than only being trained with the low dynamics-shift transitions.  For the dynamics learning, built on the observation that achieving the same next state requires taking different actions in different domains, MOBODY employs separate action encoders for each domain to encode different actions to the shared latent space while sharing a unified representation of states and a common transition function. 
We further introduce a target Q-weighted behavior cloning loss in policy optimization to avoid out-of-distribution actions, which push the policy toward actions with high target-domain Q-values, rather than high source domain Q-values or uniformly imitating all actions in the offline dataset.  We evaluate MOBODY on a wide range of MuJoCo and Adroit benchmarks, demonstrating that it outperforms state-of-the-art off-dynamics RL baselines as well as policy learning methods based on different dynamics learning baselines, with especially pronounced improvements in challenging scenarios where existing methods struggle.

Published as a conference paper at ICLR 2026
MOBODY: MODEL-BASED OFF-DYNAMICS OFFLINE REINFORCEMENT LEARNING
Yihong Guo'
Yu Yang
Pan Xu²
Anqi Liu'
Johns Hopkins University
Duke University
(yguo80, aliu. jhu edu
(yu-yang, pan, pan.xu)aduke.ee.. @duke. edu
ABSTRACT
We study off-dynamics offline reinforcement learning, where the goal is to learn a policy from offline source and limited target datasets with mismatched dynamics. Existing methods either penalize the reward or discard source transitions occurring in parts of the transition space with high dynamics shift. As result, they optimize the policy using data from low-shift regions, limiting exploration of high-reward states in the target domain that do not fall within these regions. Consequently, such methods often fail when the dynamics shift is significant or the optimal trajec- tories lie outside the low-shift regions. To overcome this limitation, we propose MOBODY, a Model-Based Off-DDnamics Offline RI. algorithm that optimizes a policy using learned target dynamics transitions to explore the target domain, rather than only being trained with the low dynamics-shift transitions. For the dynamics learning. built on the observation that achieving the same next state requires taking different actions in different domains, MOBODY employs separate action encoders for each domain encode different actions to the shared latent space while sharing a unified representation of states and common transition function. We further introduce a target Q-weighted behavior cloning loss policy optimization to avoid outtt--diitributiin actions, which push the policy toward actions with high target-domain Q-values, rather than high source domain Q-values or uniformly imitating all actions in the offline dataset. We evaluate MOBODY on wide range MuJoCo and Adroit benchmarks, demonstrating that outperforms state-of- the-art off-dynamics RL baselines well as policy learning methods based on different dynamics learning baselines, with especially pronounced improvements in challenging scenarios where existing methods struggle.
INTRODUCTION
Reinforcement learning (RL) (Kaebbling et al. 1996 Li 2017) aims to learn a policy that maximizes cumulative reward by interacting with an environment and collecting the corresponding rewards. While RL has led to impressive successes in many domains, such as autonomous driving (Kiran) let al. 2021) and healthcare (Lee et al.) 2023). it faces significant constraints on interaction with the environment due safety or cost concerns. One solution to learn a policy from pre-collected offline dataset (Levine et al. 2020). Still, when the offline dataset is insufficient, data from another environment, such simulator with potentially mismatched dynamies, may be needed, but requires further domain adaptation. In our paper, we study specific type of domain adaptation in RL, called off-dynamics offline RL (Liu et al., 2022 2024a Lyu et al. 2024b), where the simulator (source) and real/deployed (target) environments differ in their transitions. The agent is not allowed to interact with the environment but only has access to offline data that is pre-collected from the two domains with mismatched dynamics and trains policy with the offline data.
Existing works on off-dynamics offline RL solve the problem by 1) reward regularization methods (Liu et al. 2022 Xue et al. 2023 Wang et al. 2026) through the state visitation frequency or estimation of the dynamics gap or 2) data filtering methods (Xu et al. 2023 Liu et al., 2024a Wen (et al. 2024) that penalize or filter out source transitions with high dynamics shift. As result. the policy is mostly optimized with the transitions from the low shift regions, limiting exploration of high-reward states in the target domain that do not fall within these regions. Consequently, such

---

Published as a conference paper at ICLR 2026
methods often fail when the dynamics shift is significant or the optimal/high trajectories lie outside the low-shift regions. So we wonder, can we directly optimize the policy with the target transition, instead of only the low-shift regions to allow for more exploration of the high target reward and large shift region?
Motivated by this, we propose a Model-Based Off- liffDDyamiis RL. algorithm (MOBODY) that learns target domain dynamics through representation learning and optimizes the policy with exploratory rollout from the learned dynamics instead of only the low-shift region data. Existing dynamics learning methods, such as learning with limited target data, learning with combined source and target data, and pretraining on source and finetuning on the target domain, are infeasible in the off-dynamies RL setting due to the intrinsic dynamics difference in this problem. This is because 1) the dynamics learned from the combined dataset is not the accurate target dynamics, but the dynamics resemble the source one as the source transitions dominate the dataset, 2) the pretrain-finetur method still doesn't 't capture what is the difference between source and target dynamies using the same dynamics model, but only tries to learn the target domain based on the source transition.
To learn the target dynamics, we leverage shared structural knowledge across domains, such as the high-level robot motion and position in a robotics task, while employing separate modules to account for domain-specific dynamics differences. Specifi- cally, we observe that achieve the same next state starting from the same state, different actions are re- quired in two domains. Based on this, we propose learn separate action encoders for the two dynamics to encode actions into a unified action representation. and also learn a unified transition and state encoder to map the unified latent state and action representation to the next state. And such shared representation and transition functions can be learned with the auxiliary of the source data through representation learning. In this way, MOBODY learns separate transition func- tions for two domains but utilizes the source data to provide shared structure knowledge regarding the transitions. As shown in Figure I MOPO that di- rectly learns dynamics with combined source and target data significantly underperforms MOBODY, which is specifically optimized learn the target dynamics.
Norallood Scares
Target Gravity x5
40
Target Gravity x0.1
60
40
20
20
DARA
MOPO
MOBODY
DARA
MORD
MOBODY
Figure 1: Comparison between DARA (Liu
et al., 2022) (a SOTA model-free reward reg- ularization method for offline off-dynamics RL), MOPO (Yu et al. 2020) (a vanilla model- based offline RL), and MOBODY on two Mu- JoCo tasks. We show that 1) the model-free method DARA receives low reward compared with model-based MOBODY due to a lack of exploration in the target domain, and 2) MOPO fails as it cannot learn a good transi- tion for exploration with combined source and target dataset.
We further propose a practical and useful target Q-weighted behavior cloning regularization in the policy learning to avoid -uitof-distributionn and high source Q value (but low target Q value) actions, inspired by the advantage-weighter regression (Peters & Schaal 2007, Kostrikov et al. 2021a). The vanilla behavior cloning loss (Fujimoto & Gu 2021) will push the policy to favor the action in the source data, but the action in the source data might not perform well in the target domain due to the dynamics shift. To overcome this issue, the target Q-weight behavior cloning loss regularization will up-weight action with the high target Q value. And we empirically validate the choice of the target-Q weight BC loss.
Our contribution can be summarized as follows:
We propose a novel paradigm for off-dynamics offline RL, called model-based off-dynamics offline RL, that can explore the target domain with the learned target transitions instead of optimizing the policy only with the low-shift transitions.
We propose novel framework for learning the target dynamics with source data and limited target data by learning separate action encoders for the two domains while also learning a shared state and the transition in the latent space. We also incorporate a target Q-weighted behavior cloning loss for policy optimization that is simple, efficient, and empirically validated for off-dynamics offline RL settings.
We evaluate our method on MuJoCo and Adroit environments in the offline setting with different types and levels of dynamiessss shifts and demonstrate the superiority of our model with an average 44% improvement over baseline methods on the gravity and friction settings and 25% on the kinematic and morphology shift settings.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
Off-dynamics oftfine reinforcement learning. We consider two Markov Decision Processes (MDPs): the source domain Marc (S, A, R. Pure 7) and the target domain Mig (S, R. Pig. 7). The difference between the two domains lies in the transition dynamics p, i.e., Parc # Pug or more specifically, (s 3.a) * Pug(s'
s. a). Following existing literature on off-dynamics RL
(Eysenbach et al.) 2020 Liu et al., 2022; Lyu et al., 2024a] Guo et al.] 2024 Lyu et al.] 2024bb Wen et al., 2024). we assume that reward functions are the same across the domains, which is modeled by the state, action, and next state, i.e., (s, a, 8') Fing (sgg( a, 8'). The dependency of the reward on s' is well-justified in many simulation environments and applications, such as the Ant environment in MuJoCo, where the reward is based on how far the Ant moves forward, measured by the change in its x-coordinate (i.e., the difference between the x-coordinate after and before taking action). The goal is to learn policy ı with source domain data (s, a, s', rrs and limited target domain data (s, 0, "') mig that maximize the cumulative reward in the target domain max, Pap I2x, 7 (st, ).)] In the offline setting, we are provided with static datasets from source and target domain Dae {(s, 0, 8', rr)ee and Dur- = {(s, a, r'gr), J. which consist of the
transitionsstraeecoorii collected by some unknown behavior policy. Note that in the off-dynamics setting, the number of transitions from the target domain is significantly smaller than the source, i.e.,
|Dugl < | l. and normally the ratio [Daal can vary from 10 to 200. In our paper, we follow the
Den
ODRL benchmark (Lyu et al. 2024b) in which the ratio is 200.
Model-based offiine reinforcement learning. Model-based RL learns transition function T(s', ros,rss,a a) by maximizing the the likelihood T maxy Ea [log rss, a)]. Then, the al- gorithm rolls out new transition data to optimize the policy and take u(s, a) as the uncertainty quantification to obtain conservative transition, i.e., (s, a, a, au(ss, a)). The policy with offline data Domine and online rollout (s, a, a,,' f au(s,a) However, different from traditional model- based offline RL, we only have very limited target domain data and source data with dynamics shift. There is no existing model-based solution for off-dynamics RL, which calls for novel methodology development both in dynamies learning and policy learning.
rrss,)),,
Detailed discussions of related work are in Appendix A due to space limit.
3 MOBODY: MODEL-BASED OFF-DYNAMICS OFFLINE REINEORCEMENT LEARNING
In this section, we present our algorithm. MOBODY, for the off-dynamics offline RL problem setting. We first present how we learn the target dynamics with very limited target domain data Dug and source do- main data Dec. Secondly, for policy learning, we in- corporate a target Q- weighttd behavior cloning loss to regularize the policy, where the target Q value is learned from enhanced target data, including reward regularized source data, target data, and rollout data from learned dynamics. The algorithm summarized in Algorithm?
(x((s)
Farvies
Looober
Opak Siraatio LAn
Figure 2: Architecture of the dynamics model. MOBODY encodes the state with Φg and state action with vi, outputs the next state through Φr, and learns the dynamies for both domains by transition loss shown in purple double arrow < It learns the state action representation by matching the state action representation Zso with the next state repre- sentation z,' through encoder loss shown in the green double arrow + and the state repre- sentation through cycle transition loss shown in orange double arrow e
3.1 LEARNING THE TARGET DYNAMICS
Decomposition of the dynamics. In general, the dynamies can be modeled as s' = a(s, a) or 8' (s. a) and s' (", a) for the two domains. Although the dynamics are different, the transitions share some structured knowledge that can utilize. Also, from another perspective, for the two domains, to achieve the same next state, different action is required, i.e., (s, a sae and (s, arg. s') Irg. Based on this, we propose using separate action encoders encode actions from the two domains into the
3

---

Published as a conference paper at ICLR 2026
shared latent space. So the source and target domains can share unified representation of states and a common transition function with the latent action.
We define the 2, as the state representation, zsu and th as the state-action representations from the action encoder for source and target dynamics, respectively. Specifically, we model the state and state action representation through 2, = (s), Zea = Vug (2, a) and 2 = (2, a), so that we can obtain a separate state action representation for two domains. So, with the learned representation 2, and z we have the dynamics modeled as s' or( ((z,,))). where Ør the transition function. For simplicity and to reduce the model parameters, we choose to directly add the state and state action representation together and feed into the transition function 8' = ΦT + (b(,,, a)). Note that this additive term 2, + (b(,, a) is also widely adopted in the implementation of the model-based RL., where the transition is modeled as = 8 + [(s, a). We show the flow of the dynamics learning And our dynamics model is:
component in Figure 2
source dynamics :2, (b(s). Rea 2s + (ss 8' (r((ss); target dynamics :2, dg(s), Rza =2 + (2,
(1)
(=sa);
(2)
where dg is the state encoder, ØT is the transition, and there and t'mn are the state action encoders for source and target, respectively. Equation | and Equation (2) show that we use different modules (action encoder Visce and bugg for the source and target domains, but with shared state representation from state encoder Φg and unified transition function Φr. We now discuss how representation learning techniques, utilizing several loss functions, enable us to learn representation and dynamics.
Transition Loss. The transition loss minimizes the Mean Squared Error of the predicted next state and the ground truth next state as shown in a purple two-way arrow in Figure 2 The goal of the transition loss is to learn the shared transition knowledge ør using both source and target data.
sDc dyn
||s 0r(2. + the (2., a)) = E 1|s' - (r + (2g a))l?
(3)
Encoder Loss (Learning separate action encoder V'scc and Vug)). The Φr can map the latent state action representation to the next state for both domains, we use encoder loss to learn the separate action encoders for the two domains to map different actions to the unified latent space that served as the input to the drr. Specifically, we adopt general assumption in representation learning that the representation of the state action should be close to the next state (Ye et al.. 202T Hansen et al., 2022b), where the predicted representation of the current state-action pair (F... a) incorporates the transition information encoder to further encode the difference of the dynamics information for the two domains, thereby improving the efficiency of learning the dynamics model. The encoder loss is formulated as:
be close to the next state representation (s') This encourages the action
N |l | - (z+ + (2,, I) L Tuop N
N M
N ||| |
- (z+ + Vin a))l", (4)
2g' (s') is the next state representation encoded with &F and IxIx is the stopping gradient. Here, N is the batch size. The encoder loss is shown in a green two-way arrow in Figure 2
Cycle Transition Loss (Learning shared Φr and ΦT). To further improve the state representation quality and avoid mode collapse in the encoder loss, we include "'yycle transition loss" through VAE-style (Kingma et al., 2013) learning. The dynamics function maps the state action to the next state through the state action representation. Then, from one perspective, by setting vi to O, the dynamics only input the state into the dynamics learning framework, and no action will be taken. The output of the dynamics will be the same state, i.c., (s, 0, s), which is the same for two domains. So when the v is set to 0, the state is predicted as: 3 = ())r)s)) + 0). Then we can explicitly learn the state representation with the state in the offline dataset by minimizing: (()T(()(())) - a||2- From this perspective, we can view dg as an encoder and ΦT as a decoder, and we propose using a Variational AutoEncoder (VAE) (Kingma et al. 2013) to learn the state representation.
Let 2g be expressed as 2, = Pep (s) + ae (s) O c, with c ~ NN, I) and (d and dør are the output of state encoder network Φg. Let d_ be the dimension of the latent representation, the loss for learning the state representation is:
Lyyaee
N
E (pis + +o - Logo 1) N ΣN ||s, 3:ll2-
(5)
2N
The cycle transition loss is shown in orange two-way arrow in Figure 2
Unlike previous VAE-based dynamics learning methods, which are not tailored for off-dynamice RL, we introduce cycle transition loss alongside the encoder loss to jointly learn state representations
4

---

Published as a conference paper at ICLR 2026
and shared transition functions across domains, rather than just learning state representations. The VAE representation also mitigates mode collapse that arises when trained solely on the encoder loss. The decoder, serving shared transition function, maps the unified state-action representation from separate action encoders to the next state, providing additional supervision signals for learning cross-domain dynamics. In conclusion, our method learns the unified transition function dr for both domains, while using the ehace and Ving to learn the distinct information of the two dynamics.
Reward learning and uncertainty quantification (UQ) of the learned dynamics. Given that tuple and the reward function is the same across
the reward is modeled as a function of (s, a,,') 8') domains, we learn the reward function f(s, a, s') as a function of the (s, a, s') tuple with the combined source and target dataset through the MSE loss Lieward D.D.. [r(ss, a, s') - f(s, a, s')]
DED [r(s, a, s') f(s, a, 3²)]²² where s' is the predicted next state. Here, we use both the true next state and the predicted next state from the dynamics model learn the reward model, as during inference, we do not have the true next state and only have a predicted next state. This is further described in the Appendix B Also, we follow the standard model-based approach (Yu et al. 2020) for the UQ of the dynamics, by penalizing the estimated reward f with the uncertainty in predicting the next state: f(s, a 8') = f(8, a, s') - Bu(s, a) where u(s. a) the uncertainty of the next state and B is the scale parameter. We refer to the details in Appendix B
To summarize, the dynamics learning loss is:
min D Laya + ED, - ti + DEDmuDDy! + Avp (Legeee + [xep]]:
(6)
where Arep scalar controlling the weight of the representation learning term and set to be in the experiments, we notice there is no significant performance difference with different Acep-
And the representation loss is summing the source and target loss: Leyele = L Late and
Lycle
Lrep
L sic + Lipp. We summarize the dynamics learning algorithm in Algorithm 1
rep
3.2 POLICY LEARNING WITH THE TARGET-QU Q- WEIGHTED BEHAVIOR CLONING LOss
After we learn the target dynamics, we perform model-based offline RL training. During the policy optimization, we roll out new target data from the learned target dynamics with the current policy and state in the offline data and keep the rollout data in the Diake- Also, we want utilize the source data to optimize the policy. We follow the previous work by DARA (Liu et al. 2022) on off- off-dynamics offline RL. This approach first performs reward regularization on the source data, which learns domain classifiers p(trg 8, a, 8') and p(trgls, a) to penalize the reward of large shift in the source
data: TDARA (x, a) = r(s, a) - 17 log (a.(s'(a.0)
(s.a)" Details of the DARA are referred to in Appendix C.
Our enhanced target data is Deshancod = Dere_ U Dug U Drike, a combination of regularized source data, target data, and model rollouts.
Learning the Q function We learn the Q functions following standard temporal difference learning with enhanced target data:
(++ Qo- (s', Qu(s,a))
min Co- min ED an DDaae
(7)
Policy optimization with target Q-weighted behavior cloning. In offline RL, a central challenge is exploration error, as dfito--distribution actions cannot be reliably evaluated an issue exacerbated under off- off-dynamics settings. Behavior cloning (Fujimoto & Gu [022] Goecks et al. 2019) offers simple and effective regularization by biasing the policy toward actions in the offline dataset, by pushing actions close to the actions in the offline dataset. However, in off-dynamics RI., naively cloning source-domain actions can harm performance: actions in the source dataset may perform poorly in the target domain due to the dynamics shift, so vanilla behavior cloning alone in TD3-BC Fujimoto & Gu 202T) is insufficient for policy regularization.
Instead, inspired by the advantage weighted regression and the IQL (Kostrikov et al., 2021a), i.e.
(Ø) (s,,a)>D cxp (3 (Qo(s, a) Vs())) T s)}. which re-weight the log likelihood of the offline data with the advantage, we can re-weight the behavior cloning loss with the target Q
value, namely Q weighted behavior cloning loss, where the target Q value is learned with enhanced
target data, that this Q value approximates the Q value in the target domain. Intuitively, the target
5

---

Published as a conference paper at ICLR 2026
Q- wweggttdd behavior cloning loss up-weights the policy's loss with higher target Q-values, guiding the policy toward actions expected to perform better under target dynamics. The policy loss with Q weighted behavior cloning loss is:
arg min Donmol [AQ [s)s),(ss)]
Q(s, (()) I/NE, [Q(s), ((s)))
(=(s) - ag²]
Do mg mig
exp
(8)
where the À
I/N _. [Q(s,a]] is the scaler À that balance the behavior regularization error and Q loss and a a hyper parameters. We empirically validate the choice of target Q-weighted instead of AWR style loss in Section 4.3 and Appendix C.4.2 We summarize the MOBODY in Algorithm 2 the Appendix
4 EXPERIMENTS
In this section, we empirically evaluate MOBODY in off-dynamics offline RL settings using four MuloCo environments from the ODRL benchmark: HalfCheetah--2, Ant-v2, Walker2d-v2, and Hopper-v2 and manipulation tasks in Adroit: Pen and Door. We also perform comprehensive ablation studies justify the importance of each component of MOBODY.
4.1 EXPERIMENTAL SETUP
Environments, Tasks, and Datasets. We evaluate MOBODY on the MuloCo and Adroit envi- romments from the ODRL benchmark (Lyu et al. 2024b). For the MuJoCo environment, we set the source domain unchanged and consider several types of dynamics shifts for the target domain, 1) gravity and friction, each scaled at four levels: {0.1, 0.5, 2.0, 5.0} by multiplying the original values in MuJoCo, and 2) kinematics and morphology shift, each is achieved by constraining the rotation angle ranges of certain joints modifying the size of specific limbs or the torsos of the robot. We also consider the Adroit task with kinematics and morphology shift, scaled to medium and hard shift levels, to demonstrate that our method applies to wide range of environments and shift types/levels. We use the medium-level offline datasets collected by the ODRL benchmark, which uses SAC-trained behavior policy tuned to achieve about 50% of expert performance. The target dataset is then collected through rollout trajectories until 5,000 target transitions are reached. Also, the source data contains I million transitions.
We evaluate the performance with the Normalized Score, defined as: normalized__se _scoreddeeceeeeeee = random expperiscoore score-random, seoodoo_sooreeeee 100, where the random_scon is achieved by the random policy and the expert_scon score achieved by the SAC (Haarmoja al. 2018) trained to the expert level in the target domain. We also conduct hyperparameter and computational cost analysis in the Appendix C.4 demonstrate that our method not overly sensitive hyperparameters.
Baselines. We compare MOBODY against model-free, model-based, and off-dynamics offline RL baselines. For model-free methods, we use IQL (Kostrikov al. 2021a) and TD3-BC (Fujimoto & Gu 202T), trained directly on the combined offline dataset of source and target transitions, without any modification tailored off-dynamics settings. For model-based offline RI., we adopt MOPO YYu et al.] 2020): instead of training dynamics only on the target domain (which performs poorly in our setting), we follow prior off-dynamics work (Eysenbach et al., (0200) and train MOPO's dynamics model and policy the combined source+target dataset. We further include representative off-dynamics offline RL methods DARA (Liu et al. 2022). BOSA (Liu et al. (024a), and SRPO (Xue et al.) 2023) and REAG (Wang al. 2026). Finally, compare MOBODY with alternative dynamics learning strategies in Section 4.3.2
4.2 MAIN RESULTS
Results on MuJoco gravity/friction shift. Table I we show the detailed results and highlight the best and second-best scores of the MuJoco gravity and friction shift problems. In the last row of Table I we sum the normalized scores in total. Our proposed MOBODY receives 44% improvement the best performing baselines, REAG, and performs the best second best in 28 out of 32

---

Published as a conference paper at ICLR 2026
Table 1: Performance of MOBODY and baselines on four MuJoCo tasks under medium-level offline dataset with dynamics shifts in gravity and friction (levels 0.1, 0.5, 2.0, 5.0). Source domains remain unchanged; target domains are shifted. We report normalized target-domain scores (mean + std over three seeds). Best and second-best scores are highlighted in cyan and light cyan, respectively. MOBODY receives 44% improvement over the second best baseline REAG.
Level
BOSA
IQ1.
TD3-BC
MOPO
REAG
SRPO
MOBODY
9.31
9.62 6.27
6.90
0.22
12,90 1.01
16.14
$2.04 1.05
14.18
HalfChertah Gravity
5.68
44.23
6.38
2:220
46.11 1.53
40.50
27
0.01
3.60
10:40 57.75 18.92
31,85
32:21-11..77
7.35
5.0
17:95
11:97
44.00 23.13
13.75 14:11
27.67
17.01
71.31 2,80
-2.33
80.05
1.21
12.58
8.83
26.20
8:06
28.32
20.69
46
9.74
17.36 1.73
37.53
2.49
HallChhetth
08:00
0.35
00:80
0.64
9.91
$4.98
5.91
64.89 2: 3.04
06.50 4: 0:99
109:18:
1:18::22:11
2.0
46.33
0.57
46.03
2.04 64.06 6.78
43.51 0.74
42.93 8.88
46.25
2.36
37.74 2:35
75.19 + 1.54
50,02
5.0
44.07 0.07
35.83
1. 6:65
42:30 10:22
40.06 1. 7.87
25.74 3.24
5.10
1.06
50.20
4.00
0.1
25:58
7.21
12.53 1:11
13:23 2.61
8.03 1.20
11.03 1.24
15.75 1.17
13.75 1.81
37.00 1. 2.22:2
Ant
0.5
19.03
4.41
10.00 2.00
12:01
2.85
12.28
1. 3.58
9:03
1.35
13.25
0.86
1.02
2.7.3
37-44
1. 2.70
Guavity
2.0
61.77
1.52
37:17 11.96
34.04 =: 4.122
35.48 +: 22222
36.66 =:: 0.82
41.25
#: 1.72
4.37
2.2.08
40.88 + 1.71
5,0
41.94
a 0.329
$1 01:35
6.37 = 0.45
28.97 4. 5.90
31.01 1: 0.30
49.35
A: 2.61
8.45 4: 1.24
65.45 3.28
0.1
68.95 + 0.71
55.06 11:46
40:20 =: 2.56
49.6 4 5-99
56.12 =: 0.24
54.13 d: 0.56
2.55
8.45
5M.70 a: 0.11
Ant
0.5
50.72
5. 57
58: 28
0.60
25.21 + 7.17
$2.26 + 5.25
56.90
0.80
57.46
+ 0.65
0.57
+ 1.76
02:41
2 4.10
Frictio
2,0
20.18
8.79
19 04.84.1 3:20
22 09 8:30
15.03 0.87
17.54 1 2.47
21.28
1.7.2
10.81 1 2.09
41 4.40
5.0
9.00
1 0.88
7:75 0.25
10.06 4.10
10.89
1 3.20
7.80 1 0.12
0.55 1 0.65
11.72
1 1.86
31.17
5:57
0.1
18.75
1: 12.02
16.04 1: 7.60
36, 88 0.96
41.08 1. 10.13
20.12 1: 5.74
26,36 1. 2.62
13:67
1.1.9
65. 85
5.08
Walker2d
0.5
$0.00
+ 20.
42.00
10.72
27. 48 3.92
40.32 5.78
29: 72
16.02
05.20
+ 2.18
50.28 +: 2.34
43.77 +2882
Cravity
2.0
8.91
2.22
25.00 63
= 10.77
11.88 9.88
26.79 3.07
322 20:20:::
1:00
13.50
d: 2.38
8.22 0.82
44.32
# 4.08
50
5,25
+ 0.50
5:42 0.29
5.12 0.18
0:90 5.65
5.46 0.08
4.01 1.13
5.12
46.05
20.73
0.1
7.58 1: 1.85
5.72 0.23
20.80
24.90
27 07.99
5.05 0::::0000 0.00
10.58 0.71
0.00 0.881
28. 28.23 1: 9:13
Walker2d
0.5
20.00
06.26 3.00
18 98
4: 3.04
68.98
1.08 42.15
0.45
Friction
2.0
39.00
17.36
63.40
7.13
89
1.00
68.38 1: 1.00
72.91 1. 0.37
15.31 1 2.75
73.74 0.49
10.07
6.91
0.00
5.70
5:34
0.28
8.36
27. 78
8.87
13:41
13:10 0:98
15.69 6.00
49
23.40
31.11 1.80
17,02
1.06
36.26 1.50
Hopper
12.77
87
23.92
12.86 0.18
2,06
+: 8,00
38.57
Gravity
2.57
16.08
14.05
1.60
23.70
5.0
7,30 1: 0.13
11.16
9.08
1.15
7.77 0.31
7.00
8.11 0.97
7.48 0.51
0.00
25.55
2.09
16.04 5:37
1:
26.
33.08 2.53
0.85
51.19
Hopper
25.22
8.48
23.56
1:68
19:00:= 15. 45
12.01 1: 3.96
26.94 a: 2.86
38:10 3.312
18:41 1.31
41.34 0.49
Priction
2.0
10.32
0.05
0.06
0.80 0.20
10.99 + 0.76
10.15 0.08
10.20 0.30
0.71
11.00 0.14
5.0
7:00
+ 0.06
7.93 0.001
1.00 7.80
7.68 0.19
7.80=00000
0.36 8:20
7.76 0.35
8.07 0.04
Total
875.88
901.62
779.14
803.22
800.02
936.13
647.01
1427.20
tasks. Also, MOBODY outperforms the baseline more in the large-shift setting, demonstrating the effectiveness of our method for exploration and the dynamics learning and the suboptimality of conservative reward regularization or data filtering methods.
MOBODY improves more when the dynamics shift is larger. Additionally, in larger shift sce- narios, such as SalfCheetah-Friittonn-- 1, Ant-Friction-5.0, and Walker2d-Friction-- MOBODY achieves significant improvement over baseline methods, which receive very low rewards in the target domain. We also summarize the performance comparison under different shift levels in Figure 5 in Appendix C.3 Existing methods, DARA, BOSA, SRPO and REAG, fail in large shift settings as the reward regularization methods are mainly trained with source data with regularization, resulting in optimizing the policy with the low dynamics-shift transitions and cannot adapt the large shift target domain, as we mentioned earlier. Thus, such methods lack exploration of high-reward states in the target domain that do not fall within these low dynamics-shift regions, which is more frequent when shift is large.
Results on MuJoco kinematics/morphooogy shift. We also conduct experiments on MuJoco and Adroit with kinematics and morphology shift. Due to page limit, we summarize the results in Figure B by summing the normalized score across different tasks. We observe that MOBODY receives a higher overall score. We do not include error bars because the results are aggregated across many tasks, and a single standard deviation not well-defined at this level of aggregation. We also present all the experimental results for each task in Table and Table in Appendix C.3 showing that our method performs the best in 29 out of 40 tasks and achieves an overall 25% improvement in all tasks.
DARA and BOSA do not have significant improvements compared with IQL. The DARA reward augmentation term, based on a KL divergence between source and target dynamics, can become ill-defined when their supports barely overlap, destabilizing training and sometimes making DARA worse than IQL. For BOSA, relying on a target dynamics model trained only on 5,000 target transitions makes accurate dynamics learning difficult, thereby degrading performance.
In a few settings, MOBODY slightly underperforms SRPO or other baselines. SRPO assumes optimal policies across dynamics often induce similar stationary state distributions. When this holds
7

---

Published as a conference paper at ICLR 2026
in some tasks, it yields strong performance there, but bad performance when the assumption is violated. However, MOBODY outperforms SRPO on most tasks, indicating better robustness under broader dynamics shifts. In the remaining MOBODY-underperfomm baseline is small (less than 1.7%), typically either because all methods fail and achieve very low rewards (e.g., Hopper- Gravitt--aa (e.g., HalfCheetah SercChetaa--riition--- Friction-O.I Dalker2d-Friction-00 1), and additional exploration benefits from MOBODY less pronounced is these settings.
cases, the gap to the best
because baselines already perform very well under small shifts
Sesteniieeee Earazt meatee
Aeystive
Maciin
Ihatt Atmel
Lmnel
Sorr Loial
(a) Kinematic Shift
(b) Morphology Shift.
(c) Manipulation Task
Figure 3: Aggregation experimental results on MuJoco kinematic and morphology shift task, and Manipulation tasks. Our method significantly outperforms the baselines besides Manipulation medium task. Detailed results of each environment, shift type, and shift level are referred to Table and Table 5 the Appendix
4.3 ABLATION STUDY
In Section 4.3. we first conduct ablation studies on two main components MOBODY: dynamics learning and policy learning, showing the necessity of each component. In Section 4.3.2 we compare our dynamies learning with others demonstrate the effectiveness and better generalization ability. In Section 4.3.3 we then conduct ablation studies to analyze how each component contributes to the overall performance and show that the model-based rollout from our learned dynamics is the main contributor, while the target-Q weighted BC loss is auxiliary but essential regularizer that further improves AWR BC loss. In Section 4.3.4] we also conduct other ablation studies to validate the choice of target-Q weighted BC loss over the standard IQL weight.
4.3.1 ANALYSIS OF THE EFFECTIVENESS OF EACH COMPONENT
We evaluate the overall effectiveness of each component, then analyze specific design choices. For dynamic learning, assess the impact of the cycle transition loss and representation learning. For policy learning, we examine the effectiveness of the Q-weighted loss.
We first evaluate the performance of our proposed dynamics learning and policy learning by replacing the dynamics learning with the existing dynamics learning model or the policy learning with the existing offline RL algorithm. We denote the ablation studies follows:
A1: Replace dynamics learning We compare our MOBODY with variant replacing the dynamics learning with the existing model-based method. We use black-box dynamics model trained on target data only, while the policy learning follows the same method as in MOBODY. Table 2 demonstrates that the A1 variant significantly degraded compared with our proposed MOBODY algorithm in Walker2d. This indicates that only using the existing dynamics models trained on the target data is insufficient to rollout trajectories the target domain. This motivates us to propose a novel dynamics model learning method.
A2: Replace policy learning Similar the A1, we replace the policy learning with the existing offline RL algorithm. We adopt the same dynamics learning approach in MOBODY and use Conservative Q-Learning (CQL) (Kumar et al.) (2020) for policy learning. Table ( shows that our proposed MOBODY outperforms the A2 variant in Walker2d. This demonstrates that the policy learning part of our proposed MOBODY with Q Q-weight behavior cloning can better utilize the dynamics model compared with the existing method.
Then we delve into the details of the dynamics learning and policy learning part, especially our designs of the loss function and Q Q-weighting. We have the following ablation studies:
8

---

Published as a conference paper at ICLR 2026
A3: No Cycle Transition Loss Here, the dynamics model follows the dynamics learning of the proposed MOBODY, but without the cycle transition loss. We hope to evaluate the effectiveness of proposed cycle transition loss. Table iillustrates that the A3 suffers degradations compared with our MOBODY in most of the settings. This indicates that the cycle transition loss helps learn a better state representation in our proposed MOBODY method.
A4: No Q-weighted Similar to the A3, we compare our MOBODY with a variant without the Q-weighted behavior cloning loss. We keep the same dynamics learning method our proposed MOBODY and replace the Q-weighted behavior cloning loss with the vanilla behavior cloning loss. In Table our method outperforms the method without the Q-weighted behavior cloning in Walker2d. The A4 underperforms MOBODY in most of the settings except the Walker2d 2.0 level, where all settings have similar performance. This suggests that our proposed Q-weighted approach can help regularize the policy learning in the off-dynamics offline Rl. scenarios.
Table 2: Performance of the ablation study of our proposed MOBODY method. Al-A4 represent four different ablation studies detailed in Section 4.3 The experiments are conducted on the Walker2d en- vironments under the medium-leve level with dynamics shifts in gravity and friction in {0.1, 0.5, 2.0,5.03 shift levels. The source domains are the original environments, and the target domains are the envi- ronments with dynamic shifts. We report the normalized scores the target domain with the mean and standard deviation across three random seeds. The higher scores indicate better performance. More experimental results on Hopper are Table i Appendix C.4
Algorithm Ablation
Loss Ablation
Env
Level
MOBODY
A1
A2
A3
A4
0.1
55.23 i 10.22
55.43 1 5.31
35.34
10.97
19.53
4.68
65.85 1 5.08
Walker2d Gravity
0.5
35.66 1 3.11
39.98 1 1.32
30.63
1 2.92
24.44 1.91
43.57 2.32
2.0
31.94 5.32
28.58 + 5.59
34.42 3.60
47.13 2.44
44.32 + 4.58
5.0
3.56 0.79
+
4.42 1.20
46.05 + 20.73
0.1
24.34 1 10.33
25.73
2.43
21.42 1.82
19.48 1448:44 4.32
28,23 3 1 9.13
Walker2d Friction
0.5
56.31 7.17
73.23
3.73
68.53
61.38 6.844
76.96 1 1.99
60.52 5.82
71.14
2.59
67.98 6.96
73.74 6.44
5,0
4.32 + 0.85
18.32 + 2.18
5.42 = 0.82
7.89 = 1.33
27.38 3.87
4.3.2 COMPARISON AMONG DIFFERENT DYNAMICS LEARNING APPROACHES
As model-based method, MOBODY learns target dynamics that generate higher-quality transitions and yield lower estimation enror. We compare against three dynamics-learning baselines: (1) target- only training, (2) combined source+target training, and (3) pretrain-finet finetune (pretraining on source, then finetuning on target). For fair comparison, all methods share MOBODY'S architecture but use a single action encoder and omit the cycle-transition loss. We evaluate the learned dynamics by the rollout MSE under the MOBODY policy at 1M training steps.
Table Performance comparison using different dynamics learning models. First row: evaluation MSE of the rollout trajectories using different MOBODY policy, second row: normalized score of the policy. We see that MOBODY outperforms the baseline dynamics learning methods in both dynamics learning and overall performance.
Metric
Task
Trained only target data
Combined data
Pretrained-finetun
MOBODY
Walker2d-frietiont
2.23 = 0.26
1.96 = 0.68
2.21 = 0.19
1.25 : 0.39
MSE
Ant-friction-0.5
Salker2d-gravity-lll
2.11 0.48
0.51
2,01 0.24
1.87 + 0.32
2.14 0.19
2.32 4: 0.23
1.93 + 0.34
1.88 : 0.18
Ant- gravity-0.5
1.57 + 0.39
1.53 + 0.39
1.73 0.43
1.46 + 0.26
Walker2d-ffictioon-0..
56.31 +: 7.17
41.38 5.12
62.93 = 5.43
76.96 1: 1.99
Normalized Score
Walker2d 2alker2d-gravit--0.55
42.13 3.98
3.12
-itffriccion--.5
48.13 + 4.43
46.23 4 6.85
51.09
1.93
62.41
+: 4.10
Ant- gravity-055
28.32 + 3.87
31.39 + 3.80
29.69
7.23
37.44
2.79
Table reports both policy performance and rollout MSE for different synamics-learning strategies, and shows that MOBODY consistently outperforms all baselines in both MSE and reward. This is

---

Published as a conference paper at ICLR 2026
mainly because: (1) the target-only dataset is too small to learn accurate dynamics, (2) training on combined source+tareet data yields a model whose dynamics lie between the two domains rather than matching the target, and (3) the finratunn paradigm, while effective in supervised domain adaptation, is less suitable for off-dynamics RI, where the conditional next state (s. a) fundamentally differs across domains. In contrast, MOBODY explicitly learns shared structure while using separate action encoders to capture the dynamics differences between source and target.
4.3.3 MODEL-BASED ROLLOUT IS THE MAIN DRIVER OF IMPROVEMENT, AND TARGET-Q Q WEIGHTED BC IS ESSENTIAL
To demonstrate that the main performance gains come from dynamics learning. model-based rollouts, and ex- ploration, we compare AWR/IQL-style policy learning with and without MOBODY's rollouts. In Figure + we report results for four variants: (B1) IQL-style policy learn- ing without model-based rollouts, (B2) AWR-weighted BC with MOBODY rollouts, (B3) Target-Q--eiihted BC without rollouts, and (B4) the full MOBODY method.
B1: AMi wwight, wio rollutt 18Q1)
500
82: AWiit BC wiight, wi rollele
B3: Twrget-Q weight, win rollout
Target-Q wwight, rollout (MOBOCY)
450
400
By comparing (B1) vs. (B2) and (B3) vB.( differ only in whether model-based rollouts are used-we we observe substantial gains from incorporating MOBODY's rollouts, highlighting the effectiveness of our learned dy- namics. In contrast, comparing (B1) (B3), which differ only in the BC weighting (IQL vs. target-Q), shows no substantial improvement from using target-Q-weighte BC alone, indicating that this auxiliary term is not the main source of performance gains.
350
300
Figure 4: Comparison of different BC losses with and without dynamics learn- ing shows that MOBODY's performance gains primarily stem from its novel dy- namics learning and model-based roll- outs.
However, we still want to highlight that the improvements arise from the full algorithm working in concert rather
than from any single component in isolation, and each component is important. As further supported by Section 4.3.I where removing the BC weight for the MOBODY will lead significant drop.
4.3.4 EMPIRICAL VALIDATION OF THE TARGET-Q WEIGHTED BC LOSS
In Figure comparing (B2) and (B4), which differ only in the BC weighting, shows a clear performance gain for the target- Q- Q-weighted BC used in MOBODY. AWR well-suited for policy improvement (as in IQL), but might be less effective as a regularizer in off-dynamics RL: it is more important to bias the policy toward actions with high target-domain Q-values than high advantages, and when advantages shrink toward zero, it provides little supervision for the BC loss. In addition, the target-Q- Srget---weiihhed BC loss has a simpler form, whereas AWR/IQL requires training an extra value network, complicating optimization in the model-based setting. These results empirically support our choice of target-Q- Q-weighted BC, and we provide a more detailed comparison with AWR/IQL in Appendix C.4.2
5 CONCLUSION
We study the off-dynamics offline reinforcement learning problem through a model-based offline RL method. We introduce MOBODY, a model-based offline RL algorithm that enables policy exploration in the target domain via learned dynamics models. By leveraging shared latent representations across domains, MOBODY effectively learns target dynamics using both source and limited target data, Additionally, we propose a Q-weighted behavior cloning strategy that favors actions with high target Q value, further improving policy learning. Experimental results on MuJoCo and Adroit benchmarks demonstrate that MOBODY consistently outperforms prior methods, particularly in scenarios with significant dynamic mismatches, highlighting its robustness and generalization capabilities. Our method shows the potential of data augmentation in policy learning with carefully learned dynamics model. Future work includes further investigation on improving the dynamics learning as well as investigation sparse reward and goal-conditional RL settings.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Our codes are available at: http://hhhhh.oooolyyoooooooooo/d
Learnnng The implementation of the method based on the ODRL bench- mark repository {Lyu et al. (0044)) which provides the comprehensive dataset and baseline method for evaluation, For our algorithm, we provide detailed information on the training loss for the dy- namics learning and the policy optimization in the main text as well as the Algorithm [ dynamics learning and Algorithm for policy optimization Appendix B We also provide hyperparameter analysis and oile-of-thumbb hyperparametens in Appendix C.4. as well as the hyperparameters and model architecture that we used for tuning in Table
ACKNOWLEDGMENTS
This work is partially supported by grant from the JHU Center for Digital Health and Artificial Intelligence and grant from Open Philanthropy. YY and PX are supported in part by the National Science Foundation DOMS-232310 122) and the Whitehead Scholars Program at the Duke University School of Medicine. We thank anonymous reviewers for their constructive feedback on this document.
REFERENCES
André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, David Silver, and Hado van Hasselt. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, volume 30, pp.
2017.
Nicolò Botteghi, Mannes Poel, and Christoph Brune. Unsupervised representation learning in deep reinforcement learning: A review. IEEE Control Systems, 45(2))26 2025.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5084-15097, 2021.
Pierre Clavier, Laixi Shi, Erwan Le Pennec, Eric Mazumdar, Adam Wierman, and Matthieu Geist. Near-optimal distributionally robust reinforcement learning with general [_p norms. Advances in Neural Information Processing Systems, 37:1750-1810, 2024.
Laurent El Ghaoui and Arnab Nilim. Robust solutions to markov decision problems with uncertain transition matrices. Operations Research, 53(5):780-795, 2005.
Benjamin Eysenbach. Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arXiv preprint arXiv:2006. 13916, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 14:2013--20155 2021.
Scott Fujimoto. David Meger, and Doina Precup. A deep reinforcement learning approach to marginalized importance sampling with the successor representation. In International Conference on Machine Learning, pp. 3518 33189952.. PMLR, 2021.
Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
Vinicius G Goecks, Gregory M Gremillion, Vernon J Lawhern, John Valasek, and Nicholas R Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. arXiv preprint arXiv: (ativ::990000.. 192..9421.,,,....111111111111188 2019.
Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, and Anqi Liu. Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation. In Advances in Neural Information Processing Systems, volume 37, pp. 163226133330 2024.

---

Published as a conference paper at ICLR 2026
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Internationall conference machine learning. pp. 1861-1870. PMLR, 2018.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In International Conference on Machine Learning (ICML), pp. 8387-8406, PMLR, 2022a.
Nicklas Hansen, Xiaolong Wang. and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.0423 2022b.
Yiting He, Zhishuai Liu, Weixin Wang, and Pan Xu. Sample complexity of distributionally robust off- dynamics reinforcement learning with online interaction. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 22595-22646.. PMLR, 13-19 Jul 2025.
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data, International Conference on Learning Representations (ICLR), 2017.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neurul information processing systems, 33:
21810-2188 2020.
Diederik Kingma, Max Welling, al. Auto-encoding variational bayes, 2013.
Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE transactions on intelligent transportation systems, 23(6):4909 4926, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.0611 2021a.
Nya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations (ICLR), 2021b.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1 119111999, 2020.
Hyeonhoon Lee, Hyun-Kyu Yoon, Jaewon Kim, Ji Soo Park, Chang Hoon Koo, Dongwook Won, and Hyung-Chul Lee. Development and validation of a reinforcement learning model for ventilation control during emergence from general anesthesia. npj Digital Medicine, 6(1):145, 2023.
Chang-Hoon
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0 2020.
Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274. 2017.
Guoqing Liu, Chuheng Zhang. Li Zhao, Tao Qin, Jinhua Zhu, Jian Li, Nenghai Yu, and Tie- Yan Liu. Return-based contrastive representation learning for reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. arXiv preprint 2rXiv:2033.06666 2022.
Jinxin Liu, Ziqi Zhang Zhenyu Wei, Zifeng Zhuang. Yachen Kang, Sibo Gai, and Donglin Wang. offlie reinforcement learning. Proceedings
Beyond ood state actions: Supported of the AAAI Conference Artificial Intelligence, volume 38, pp. 13945-13953. 2024a.
12

---

Published as a conference paper at ICLR 2026
Zhishuai Liu and Pan Xu. Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation. In International Conference on Artificial Intelligence and Statistics, pp. 2719-2722 PMLR. 2024a.
Zhishuai Liu and Pan Xu. Minimax optimal and computationally efficient algorithms for distribu- tionally robust offline reinforcement learning. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Zhishuai Liu and Pan Xu. Linear mixture distributionally robust markov decision processes. In The Thitty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL
/ /ttpr///eppeeeeeeeeeeemme net/ forum?id Gd-MRdA----G--1I11
Zhishuai Liu, Weixin Wang, and Pan Xu. Upper and lower bounds for distributionally robust
off-dynamics reinforcement learning. arXiv preprint (tt.:24092202/// 2024b.
Miao Lu, Han Zhong. Tong Zhang, and Jose Blanchet. Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Jiafei Lyu, Chenjia Bai, Jingwen Yang, Zongqing Lu, and Xiu Li. Cross-domain policy adaptation by
capturing representation mismatch. arXiv preprint 20:::2205.55339 2024a.
Jiafei Lyu, Kang Xu, Jiacheng Xu, Jing-Wen, Yang, Zongzhang Zhang, Chenjia Bai, Zongqing Lu,
Xiu Li, et al. Odrl: benchmark for off-dynamics reinforcement learning. Advances in Neural Information Processing Systems, 37:59859-5991 2024b.
Arnab Nilim and Laurent Ghaoui. Robustness in markov decision problems with uncertain transition matrices. Advances in Neural Information Processing Systems, 16, 2003.
Kei Ota, Tomoaki Oiki, Devesh K Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing in- put dimensionality improve deep reinforcement learning? In Proceedings of the 37th International Conference Machine Learning, pp. -433. PMLR, 2020.
Wendy S Parker. Ensemble modeling, uncertainty and robust predictions. Wiley interdisciplinary reviews: Climate change, 4(3):213-223, 2013.
Jan Peters and Stefan Schaal. Reinforcement learning by wewgh--eighttd regression for operational space control. In Proceedings of the 24th international conference Machine learning, pp.
745-750, 2007.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 1607-1612, 2010.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein-
forcement learning and imitation learning: tale of pessimism. Advances in Neural Information Processing Systems, 34:11702- 11716, 2021.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model- offline reinforcement learning. Advances in neural information processing systems, 35:16082-16097, 2022.
Ruhan Wang, Yu Yang, Zhishuai Liu, Dongruo Zhou, and Pan Xu. Return augmented decision transformer for off-dynamics reinforcement learning. Transactions on Machine Learning Research, 2026. ISSN 2835-8856.
Xiaoyu Wen, Chenjia Bai, Kang Xu, Xudong Yu, Yang Zhang, Xuelong Li, and Zhen Wang. Contrastive representation for data filtering in cross-domain offline reinforcement learning. arXiv preprint arXiv:2405.06192. 2024.
Huan Xu and Shie Mannor. The robustnes--pefformance tradeoff in markov decision processes. Advances in Neural Information Processing Systems, 19, 2006.
Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross-domain policy adaptation via value-guided data filtering. Advances in Neural Information Processing Systems, 36:73395-73421, 2023.
13

---

Published as a conference paper at ICLR 2026
Zhenghai Xue, Qingpeng Cai, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. State regularized policy optimization on data with dynamics shift. Advances in neural information processing systems, (1.32996--229311 2023.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34:25476-25488 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14111 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model- based policy optimization. Advances in neural information processing systems, 34:28954 14428994--2966, 2021.
Zhengfei Zhang, Kishan Panaganti, Laixi Shi, Yanan Sui, Adam Wierman, and Yisong Yue. Distribu- tionally robust constrained reinforcement learning under strong duality. Reinforcement Learning Journal, 1:17 2024.
Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 3331-3339. PMLR, 2021.
Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, and Houqiang Li. Masked contrastive representation learning for reinforcement learning. In Advances Neural Information Processing Systems (NeurIPS), 2020.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Off-dynamics RL. Off-dynamics RL aims to transfer the policy learned in the source domain to the target domain. One line of work is to regularize the reward of the source data with the target data using the domain classifier. Following this idea, DARC (Eysenbach et al. 2020) and DARAIL (Guo et al. 2024) solve the off- off-dynamics RL problem in the online paradigm, while DARA (Liu et al.]] and REAG (Wang et al. (2026) use the reward regularization techniques in the offline RL setting. Similarly, BOSA (Liu et al. 2024a) regularizes the policy by two support-constraneed objectives. SRPO (Xue et al. 2023) regularize the policy through the state visitation frequency on source and target domain. PAR (Lyu et al. 2024a) learns the representation to measure the deviation of dynamic mismatch via the state and state-action encoder to modify the reward. Another line of work is utilizing the data filter method, including the VGDF (Xu et al. 2023) and IGDF (Wen et al. 2024), which filter out the trajectories similar to the target domain and train the RL policies on filtered data. These data filtering or reward regularization methods in off- dy-amicss offline RL settings cannot explore the target domain substantially, while we propose novel model-based method that can explore the target domain with the learned largel dynamics. When no target data are available during training, an orthogonally related line of work studies robust Markov decision processes (MDPs), which aim to achieve reliable performance in an unknown target environment by optimizing policies against worst-case transition dynamics within a predefined uncertainty set (Nilim & Ghaoui 2003
El Ghaoui & Nilim 2005 [yengar| 2005 Xu & Mannor, 2006 Zhou et al. 2021 Liu & Xu 2024a
Lu Lu al. 2024| [Liu & Xu 2024b| Zhang et al. 2024 Clavier et al. |l.]|2224 (024] Liu et al. 2024bb He et al.
2025; Liu & Xu (2225;; ); see Liu & Xu (2025) for recent overview. However, these approaches
typically require carefully specified uncertainty sets and can be overly conservative, as they are built on the assumption that target data are available during training assumption that may not hold in some practical scenarios.
Model-based Offline RL. Model-based offline RL leverages the strengths of model-based methods in the offline RL paradigm. MOReL (Kidambi et al. 2020) and MOPO YYu et al., 2020) modify
reward functions based on uncertainty estimations derived from ensembles of models. VI-LCB
(Rashidinejad et al., (021) leverages pessimistic value iteration, incorporating penalty functions into value estimation discourage poorly-covered state-action pairs. COMBO (Yu al. (02I]) provides a conservative estimation without explicitly computing uncertainty, using adversarial training to optimize conservative value estimates. RAMBO (Rigter et al. 2022) further builds upon adversarial techniques by directly training models adversarially with conservatively modified dynamics to reduce distributional shifts. These methods are designed for one domain instead of an off-dynamics RI. setting. In this paper, we propose a novel dynamics learning and policy optimization method for an off-dynamics RL. setting.
Learning in RL. Representation learning (Botteghi et al. 2025 is actively explored in image-based reinforcement learning tasks (Kostrikov et 2002 Yarats et al. 2022 Liu et al.. 2021, Zhu et al.) 2020) to learn the representation of the image. For model-based RL, to improve sample efficiency, representation has been widely applied to learn the latent dynamics modeling (Karl
Representation
iet al. 2017 Hansen et al. 2022a). latent state representation learning (Barreto et al. [2017] 20j7/Fujlmoto Fujimoto et al. 2021). ), or latent state-action representation learning (Ota et al. 2020: Ye et al., 20221 Haasen
al. 2022b Fujimoto et al. 2023). While previous works representation learning seek to boost the performance through learning the state/state-action representation with representation constraint, such methods might not be suitable or cannot be directly applied the off-dynamics RL settings as many of them learn the representation without considering transitions or only learns single domain transitions. Thus, in our paper, we learn the shared representation of the state and transition to aid the target dynamics learning with source domain data.
B ALGORITHM DETAILS
Reward learning Note that the reward is modeled as a function of (s. s') tuple, as in many tasks, the reward is also related to the next state as mentioned in the Section [ Also, recall that the reward function in the source and target domain remains the same. Thus, we can learn the reward function with source and target domain data together via the following loss function:
Trreaad 1 [r (r(), f(s,a,
1 BD - F(0,a,8)))² (9)
15

---

Published as a conference paper at ICLR 2026
where 3' is the predicted next state. Here, we use both the true next state and the predicted next state from the dynamics model to learn the reward model, as during inference, we do not have the true next state and only have a predicted next state.
Uncertainty quantification (UQ) the transition To capture the uncertainty of the model, learn N ensemble transition models, with each model trained independently via Eq.equation
We design the UQ of the reward estimation as u(s,a) a) = max, Std ($) =
max, V E, (())), which is the largest standard deviation among all the state dimen- sions. This simple and intuitive uncertainty quantification using the ensemble model has been proven simple and effective in many machine learning literature (Parker) (003) and also model-based RL al- gorithms (Yu et al.) al.] (020). We find it sufficient to achieve good performance in our experiments by em- ploying the penalized reward i for the downstream policy learning: i(s, a, s') f(s,, a, s') -.((,,)))))
Algorithm Dynamics Learning via separate action encoders and the representation learning.
1: Input: Offline datasets arc = {(s, a, 7, s')), Dug = {(s, a, r, s')}, number of model learning steps Moodd target training frequency K.
2: Initialize: State encoder model dg, transition model Ør, source state action encoder elsres target state action encoder uug. reward model f.
Nmodel do 4: Sample mini-batch:
3: for i
5: if i% = 0 then
6:
Sample mini-batch ass,, from Dug
7:
else
8:
Sample mini-batch {(s, a, (,)'') from Dar
9:
end if
Predict the next state with Eq. equation [1 and equation 2 with mini-batch data.
11:
Optimize the dynamics with the transition loss in Eq.equation encoder loss in Eqq.quuaion
cycle transition loss Eq.equation 3 and reward loss in Eq. equation with mini-batch data.
12: end for
Algorithm 2 MOBODY: Model-Based Off-dynamics Offline Reinforcement Learning
1: Input:
Offline dataset Dare = {ss, a, r, s'}} and Dug = {(s, (,,,,, 8')), Diae U. number of model learning steps Nmodell policy training steps Npolicy-
2: Initialize: Dynamics model, policy Te. rollout length Lyollout-
Dynamics Training
3: Learn target dynamics and reward estimation: Tugs Pug Call Algorithm 1 Offline Policy Learning
4: Regularize source data Dare_ang ((s, m/rr, with DARA.
5: for 1 to Npolicy do
6: Collect rollout data from f and fug starting from state in D aar_aug and Dug. Add batch data
replay buffer Dakee
7:
Sample batch (s,a, s', r)ake from Dtake (s, a, s', r',r) from Dog and (s, a, s', r)ng from Dare_ mug. Concatenate them as (s, s',
8: Learn the Q value function with Eq.equation
9: Update policy To with Eq. equation g
end for
11:
Return: Learned policy
C EXPERIMENTAL DETAILS
C.I THE DARA REGULARIZZTION FOR SOURCE DATA USED IN MOBODY
Note that in MOBODY, we use DARA to regularize the reward the source data. this section, we introduce the details of DARA.
16

---

Published as a conference paper at ICLR 2026
DARA (Luu al. 2022 the offline version of DARC (Eyenbbach al. 2020). trains the domain classifiers to calculate the reward penalty term Ar(s, a, and regularize the rewards in the source domain dataset via:
DARA a, s') r(s, a, s') + nAr(s, as),,
where 1 is the penalty coefficient, where we set to 0.1 following the ODRL benchmark (Lyu et al. 2024b).
Estimation of the Ar. Following the DARC (Eysenbach et al., 2020 Liu ct al. 2022). the re- ward regularization Ar can be estimated with the following two binary classifiers p trg St, a.) and p(tg St, de St+1) with Bayes' rules:
p(trg) Se, ae 8441) = Pug ($141 (x)p(ss (/lrrg)(ugg)//pss ae 8641),
(10)
p(s (rg) = p(tgg 8t, ae (s). pp)gg()
(11)
Replacing the p(st, at trg) Eq. equation 10 with Eq. equation we obtain:
p(trglst, Cz. S141 (141)p((ss,s___ $1+1)
Pug ( a). a.)
p(trgl ae (s;;,
Similarly, we can obtain the (s St, ae)
pr
We can calculate the Ar(st, ar: 11+1) following:
Puz ( 18t, (8),)))))))))
Ar(st, at, St-1) = log
Pur ( 18,, an)
= log p(trpt(trrr S1, ae 86+1) - ptgg(t az) + log log p(sre|88, at, 8141) - log p (src ac).
Training the Classifier p(trg st, at) and p(trgl ag, St41). The two classifiers are parameterized bu ASs and ASSS To update the two classifiers, we sample one mini-batch of data from the source replay buffer Dam and the target replay buffer Dare respectively. Imbalanced data is considered here as each time we sample the same amount of data from the source and target domain buffer. Then, the parameters are learned by minimizing the standard cross-entropy loss:
LSAS = DED,, [log (trg St, Ot, 8141)] ED,,, [log Pasis (rrg(s,, de; St+ 1) LSA E.... [log Pos. (trgl St, at, s[+1)] ED,, [log pra. (trg st. aes st41)]
Thus, 0 (ASAS, OS^) is obtained from:
arg min LoE (Dec: Duy)
arg min|CsAs + LsA].
C.2 TECHNICAL DETAILS ABOUT BASELINE ALGORITIIMS
In this section, introduce the baselines in detail and the implementation follows the ODRL benchmark (Lyu et al. 2024b).
BOSA (Luu et al. 2024a). BOSA shows distribution shift issue might exist when learning policies from the two domain offline data under dynamics mismatch. It handles the out-of-distribution (OOD) state actions pair through supported policy optimization and addresses the OOD dynamics issue through a supported value optimization by data filtering. Specifically, the policy is updated with:
Lactor EE E a>) [Q(s, a)],
s.t. ES-D [ (s) s)] e.
Here, the e is the threshold, to is the learned policy for the combined offline dataset. The value function is updated with:
Lenii P [Q(s,a)]
+ + "f af (-1)) (Png(s'(ssa) (e) (Qo, -
17

---

Published as a conference paper at ICLR 2026
where I(.) is the indicator function, (s' a))))s arg max ('s [log Pug (s'|s, a)] is the estimated target domain dynamics, e' is the threshold.
IQL (Kostrikov et al. 2021a). IQL learns the state value function and state-action value function simultaneously by expectile regression:
Lv Pasaa (L a) (+))
where (u) = I - I(u < 0)|lu?? I(.) is the indicator function, and is the target network parameter. The state-action value function is then updated by:
L = D(saans))DD.uup (r(s, -
The advantage function A(s, a) = Q(s,a) - V(s). The policy is optimized by the advantage- weighted behavior cloning:
factrr P(o.a))Da.UD [exp(3 A(s, log
where 3 is the inverse temperature coefficient.
TD3-BC (Fujimoto & Gu 2021). TD3-BC is an effective model-free offline RL approach that incorporates behavior cloning regularization term to the objective function of the vanilla TD3, which gives:
Cactor d ..DDD.DDD (o(ss,((ss) + Pn 1(a (T()))
where
and V VER+
[ Q()),a))
the normalization coefficient.
MOPO Yu et al. 2020). MOPO a standard model-based offline policy optimization method, which learns dynamics first and penalizes rewards by the uncertainty of the dynamics. Lastly, it optimizes a policy with the SAC (Haarnoja et al. 2018). Specifically, following previous off-dynamics work (Eysenbach et al. 2020 in the online setting that applies MBPO as baseline, we learn the dynamics with the combined offline source and target data. We follow the implementation in OfflineRL-kir
DARA Liu al. 2022). We refer to the Appendix for the details. We follow the implementation in ODRL (Lyu et al. 2024b).
SRPO (Xue al.) 2023). SRPO proposes state-level regularization by leveraging the observation that optimal policies across related dynamics often induce similar stationary state distributions. We follow the practical implementation in SRPO (Xue al. 2023) train a discriminator distinguish high-value states from low-value states and then augment the reward. A simplified core objective of SRPO can be written as:
Σ (r((s, ai) s.t. (ka(d+()1s(()) Se, 1=
which, via Lagrangian relaxation, leads to a reward-shaped objective:
[(T) = BY Σ > (F(s, Allog ((Se) d, (s()) )
In practice, the density ratio
((s) d=(s)
is estimated with a discriminator (s)) trained in a GAN, yielding:
((s)
D(s)
d
88)
D(s)
REAG (Wag et al. 2026). REAG proposes that reward augmentation methods can not be directly applied to return-ondditonned supervised learning methods like DT (Chen al. 2021). It introduces
18

---

Published as a conference paper at ICLR 2026
return matching method to address this problem. We follow the description in REAG to implement the methods:
= arg min L(T) : Σ
log क (as St, ()(())))
TEDs 1=1
where ((T) Ei= Tr is the original cumulative return of trajectory т, and (b()) is the return augmentation function chosen so that ãs approximates the optimal policy in the target domain. Using dynamics-awar reward augmentation (DARA), REAG defines per-state transformed return-to-go at step as:
H
H
(g((T))
= Σ Th
+
Σ fam
Ar(sh; Uh, 8h+1).
By directly matching return distributions, REAGMY leverages the Q-values trained by CQL (Kumar al. 2020) to augment the source dataset. our experiments, we employ this variant and denote
as REAG.
C.3 ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present additional results on various types of dynamic shifts, including Kinematic shift (kin) and Morphology shift (morph), on Mujoco and Adroit, following the ODRL benchmark. We present the results in Table Table which are the detailed results of the Figure We observe that our method outperforms the baseline methods in most cases, indicating that it applicable to various types of dynamic shifts and environments.
Table 4: Performance comparison on HalfCheetah, Ant, Walker2d. and Hopper enviromments with kinematic and morphology shift. Our method performs best 25 out of 32 total tasks and receives an overall 25% improvement over baselines. We use M and H to represent the medium and hard levels of the dynamics shift.
Type
Level
BOSA
DARA
MOPO
SRPO
MOBODY
mambelligh
M
H
20.77 2 0.66
22.83 a 0.00
21.99 1.58
20.49 : 0.50
22.19 2 1.08
19.49 1 0,50
10,90 : 0.43
10.35 2 2.10
25.35 2 2.23
17.32 2 1.80
13.9620.79
99520993
216652:22
9.29 1.91
2.9.1
28.51
27.15 2: 6,90
9.20
HallChheeahh
morph-tono
kin-forrjst
H
M
H
M
17.094 1: 15.31
36.79 a 0.92
14:70 0.92
1.67 1: 0.87
31.64
S4.11
1.87 0.30
2:114
2.35
2 0.272
30.19 1:73
14.05
273 6: 1.25
5.56
2.96
31.19 2: 4.08
33.65 0.34
29.41 1: 7.88
2
12.7%
33:47
12.49
10:55 1 4.85
4.19
5.63
4.02
55.65 55.6500.06
262200114
5.0026.17
31.0850043
29 5.00
39.24
35.64
0.16 2:16200.05 0.05
4.57
1.21
13.92 :: 12.24
40.45
31.35 3.-10
18:51
1: 7.330
1.26
kin-lighju
M
H
31.72 0.17
14.92
0.00
41.27
31,50
2 9.36
2 3.16
41.77 2.66
31.10 42 9.86
31.46
15.47 1: 0.62
3: 2:31
30.35 8: 2.93
38.33
6.65
22.16:10.00
29. 22.33333
54.28
7.48
2 2.26
1.20
59.17 1:0.85
56.72
0.08
marphshalllags
73.85
50.51
1.25
45 3.60
46.60 6.24
70.06
06.32 5.29
47
52.9212.03 03
7.09
3.61
79.25 0.61
mamphaalllas
61.12
10:44
47.18 6.89
3.74
49.19 5.32
12.71 1.65
30.771:0377
11.711124
kin-onklejst
75:43 2.03
1.92
73621-034
66:22
kin-higjo
41.8513.16
morph tomo
0.1.1
4.83
12.35 1.45
2,50 0.58
1899 9:36
1.54 0.64
15.79
5
1.13
9,90
13:98
8022065
066
11.99 2.09
1.51
11.96
38.67 2.05
5.41
Walker
Lin Scorjat
mamboley
M
M
25.76
46.70
14:37
17.99
9:30
1.15
3.34
59
41.12 13.58
56.62
16:15 3.70
6.52
12.10
1.61
43.31 +: 20:48
26.34 4: 13:24
49.07 +
9:95 12:34
99 +-711-1100 13.67
55.81 1.36
13.13 1.24
9.63-1 0.91
$9.21
$7.92
44.35 6.65
19,62
0.71
20.52
41.63115.2 20
1592215575
121176676
04
51.75.24.114
50.34 2 2.92
11117 0-544
40.55 6.00
67.56
57.95 1 0.37
57.57
49.12 0,52
3.05
2.00
kin-highja
M
48 ml 14.75
47.65
27.26
61.28
S1.66
+: 2.05
14:24
15.64 8: 11:94
43.98 +: 11.54
58.28
63.26
13:79
2.06
68.11
73.52
3,00
7.93
64
64
683184115
47.22 7.85
32.42 8 2.08
69.45 4.22
78.14
2.50
morpb foot
M
12:67
10.13
0.62
0.00
32,99 =: 0.16
11:08 0.09
12.69 0.43
14.15
40.61
13.53
1.64
47.19
12.96
0.14
12/79
12.766005
11.340055
11.11 3:002
9.17 0.87
13.05
11:95
0.48
Hupper
morph tomo
11.75-1
19.58
0.08
0.05
4.60
44.67
13.29
1 0.05
0.19
49.57 0.90
1404
0.35
0.28
6987
1001+477
7.1-3.47
12.12
47.94
6.7.1
1.70
50.58
3.74
55.42.4466 66
foorjat
0.17
15.58
32:21
0:04
16.25 15:21
16.11:22.6
14885000
25
Total
964.95
863.90
1101.65
1205.70
951.01
1515.00
Figure 5 summarizes the normalized scores across all environments under different shift levels on MuJoco gravity and friction shift settings. In Figure 5a MOBODY consistently outperforms baselines under gravity shifts, with especially large gains at the more challenging and larger shift levels on 0.1 and 5.0. as MOBODY can explore more of the environment with the learned dynamics. A similar trend is observed in Figure [b] where MOBODY again outperforms all baselines, with greater improvements in the larger shift (0.1 and 5.0) compared to the smaller ones (0.5 and 2.0). Existing methods, DARA and BOSA, fail in large shift settings as the reward regularization methods
19

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison on Pen and Door tasks. Our method is the best or second best in out of 8 tasks. We use M and H to represent the medium and hard levels of the dynamics shift.
Type
Level
DOSA
[QL
TD3
BC
DARA
MOPO
REAG
SRPO
MOBODY
Pen
kis-booken-jon
1I
M
30,63
7.18
2,02
9.01
24.34 15,49
7.34 1. 3.48
6,36 6.63
1.31 1.29
33.60 3.44
9:41 6.06
57.99 2: 7,44
8.14
2.92
62.36:2240
26.30.0.49
23.41.2.39
8.1015.75
32.67 2:
13.73 1. 6.32
maspbohhrinn- finger
H
10.72 6.65
11:78 = 6.57
13:75 4.91
50.16 1.14
9.12 9.03
2.20
22.17
8.72
3.00
28.80
3.48
0.085
2.48
18.0613.522
18.38+4.59
8.7646822
7.0013.75
16,43 2: 10.46
37.90 + 1.18
Doorr
kis-beoketsjoine
H
M
25
50.64 26.87
22.04
57.43 + 12:76
56.02 + 7.74
-0.23+0000
40.12 0.02
0.01
20.18
58.22
5.20
9.91
27.00
57.45 + 9.58
1.92
60.22+5.23
18 7666++763
0.56++1.25
0.64+122 25
39.26 3.72
61.61 9.84
merph-shrink- finger
M
II
41.59
26,97
5.05
8,62
60.74
68.64
12.83
833
-0.19+-0000
0.20 1 0.02
0.1
64.22 1 7.19
50.32
4.78
67.06
$2.00
1: 1,95
1.74
62.66.1..8 1.00
0,76.10622
1.44+0998
62.88 1 5.25
63,67
9.52
Total
184.95
300,82
18.75
251.84
282.93
348.28
31,30
333.10
Performance under Different Gravity Shifts
Performance under Different Friction Shifts
Ro-sA
Total Score
MONO
MCBEOY
Total Score
MO1O
40830Y
Gravity level
Fricion level
(a) Gravity Shift
(b) Friction Shift
Figure 5: Performance of our MOBODY and baselines in different dynamics shift with various shift levels {0.1, 0.5, 2.0, 5.0). The scores are summed over all the environments (HalfCheetah, Ant, Walker2D, and Hopper) in the target domain. We directly compare the algorithms in the same dynamics shift levels. The higher scores indicate better performance. We can observe a larger improvement for larger shift cases (0.1 and 5.0).
cannot account for the large shift, as they are mainly trained with source data with regularization, thus usually receive high rewards in the source domain, but don't really adapt to the target domain, especially large shifts, as the policy gets different rewards. Also, they lack the exploration of the target domain.
C.4 ADDITIONAL ABLATION STUDY RESULTS
In this section, we provide additional ablation studies to empirically justify the design of each component and the effectiveness of MOBODY.
C.4.1 ADDITIONAL RESULTS ON SECTION
First, we present additional ablation studies results on Hopper as we mentioned in Section 4.3 Same Table we evaluate the overall effectiveness of each component (AI and A2) and then analyze specific design choices (A3 and A4). For dynamics learning, we assess the impact of the cycle transition loss and representation learning. For policy learning, we examine the effectiveness of the Q-weighted loss. We draw the same conclusion as Section 4.3
C.4.2 COMPARISON OF USING IQL WEIGHT AND TARGET Q WEIGHTED FOR BC LOSS.
We empirically justify our target Q-weighted BC loss and conduct additional experiments showing that target-Q weighted BC performs better than IQL weight.
The target Q- weightte BC loss is inspired by the advantage- advantage-weighted regression (AWR) (ostrrkov et al. 2021aa and the reward- weighted regression (RWR) (Peters et al. (200).. While the AWR is built on the RWR and we follow the idea of RWR by replacing the Monte-Carle estimation on the reward with the target value.
20

---

Published as a conference paper at ICLR 2026
Table 6: Performance of ablation study of our proposed MOBODY method. The experiments are conducted on the Hopper environments under the medium-level with dynamics shifts in gravity and friction in {0.1, 0.5, 2.0, 5.0} shift levels. The source domains are the original environments and the target domains are the environments with dynamics shifts. We report the normalized scores in the target domain with the mean and standard deviation across three random seeds. The higher scores indicate better performance.
Algorithm Ablation
Loss Ablation
Env
Level
MOBODY
A1
A2
A3
A4
0.1
14.53 + 2.81
28.07 + 4.12
33.65 = 3.21
11.54 + 1.12
36.25 + 1.50
Hopper
0.5
28.83 3.32
25.70
1.86
23.52 : 3.33
20.11 + 1.26
33.57 = 6.71
Gravity
2.0
10.64 1.92
12.32 + 5.21
10.90 = 1.29
16.40 + 4.12
23.79 + 2.09
5.0
8.12 0.69
=
8.79 4.9 0.94
8.89 + 1.01
8.06 + 0.03
0.1
26.09
4.75
35.14
24.42 2.86
20.07 10.32
51.19 + 2.56
Hopper 0.5
22.42 4: 3.32
31.31 5.08
29.26 + 6.02
27.07 3.73
41.34 + 0.49
Friction
2.0
10.64
0.32
9.41 1.03
10.31 + 0.12
8.47 + 1.13
11.00 + 0.14
5.0
8.43 = 1.32
7.52 = 0.29
8.14 = 0.91
7.55 = 1.02
8.07 = 0.04
Table 7: Comparison of using IQL-style weights for the BC loss versus the target-Q-weighte BC loss (MOBODY).
Env
Shift
IQL weight for BC
MOBODY
Halfcheetah
gravity-0.5
45.13 + 2.21
47.18
1.23
Halfcheetah
gravity-2.0
40.42 3.63
41.46 7.35
Halfcheetah
friction-0.5
64.82 + 1.53
69.54 0.48
Halfcheetah
friction-2.0
51.30 = 2.53
5..02
Walker2d
gravity-0.5
48.45 = 2.33
43.37
Walker2d
gravity-2.0
39.67 3.21
44.32
Walker2d
friction-0.5
63.47 + 5.01
76.96 1.99
Walker2d
friction-2.0
71.39 + 1.81
73.74 0.49
Here, test more standard IQL-style weighting, which is the advantage -weighte regression built on reight- regression, i.e., using IQL weights exp(3(Q(ss a) (s))). and report the results in the following table. We see that Q-weighted BC loss slightly improves the performance. The reason might be 1) training an extra value network here using model- rollout introduces more complexity and noise, leading worse performance.
Since the IQL-style weighting (AWR) is worse empirically while requiring training an additional value network (thus increasing complexity), we adopt simpler exponential weighting directly on the Q-value, inspired by IQL and reward-weighted regression (RWR). We further normalize Q by its average absolute value, following TD3-BC, because the trade-off between policy optimization and BC is highly sensitive to the scale of rewards and Q-values; this normalization yields more stable and comparable weighting across tasks.
C.4.3 COMPARISON OF DIFFERENT TARGET DATA SIZE
In Table we further provide additional experimental results with varying amounts of target data, including 500, 1,000, 2,000, and 5,000 transitions. We observe performance degradation in MOBODY as the target dataset size decreases. But this decrease is not that significant, showing our method also works well when the target data size is small.
C.5 HYPERPARAMETERS ANALYSIS AND COMPUTATIONAL COST
Hyperparameter Analysis. We conducted two hyperparameter analyses: the BC loss weight and uncertainty penalty of the model-based method in the policy learning part, as detailed in Table We that these parameters important in the performance of BC loss and need to be tuned across
21

---

Published as a conference paper at ICLR 2026
Table 8: Performance of MOBODY under different numbers of target transitions in HalfCheetah. "5,000" corresponds to the original setting in the paper and the data size used in the ODRL benchmark.
Shift-Level/data size
5,000
2,000
1,000
500
gravity-0.5
47.18 1.23
42.72 2.05
39.29 = 2.52
40.65 3.09
gravity-2.0
41.46 7.35
2 5.47
28.69 5.95
26.78
6.31
friction-0.5
69.54 = 0.48
69.62 1.08
67.56 1.52
64.14 1.90
friction-2.0
50.02 = 3.26
46.66 + 2.92
44.26 3.31
44.70 = 3.53
Table 9: Hyperparameters the policy learning. Our method is not very sensitive to the hyperparam- eters.
BC-Weight / Uncertainty Penalty
Task
0.05
76.96 +: 1.99
55.67 1 21.18
51.43 +: 19.37
W.ller2d-Friction-..5
0.1
75.64 +: 11.05
70.49 + 2.81
62.54 1: 5,49
1
75.63 2.57
82.91 + 4.43
62.50 + 6.83
0.05
67.56:+3.05
56,88=11477
65.14+2.57
Walker22-kin-fooojnnmmeeiuu
0.1
62.19:15.27
64.1713.62
66.30.10.01
1
59.33:15.15
62.69:15.00
62.56:11.09
0.05
10.96+
57.75++0.34
57.59+0.47
Walker2d-kid hinood--ii--oooo-t-aeeee
0.1
57.9311.37
56.27=1112
43.13+15.51
34.31=21.12
53.9243.74
43.74114.06
0.05
69.48114.22
60.40-14.27
66.85116.90
Walker2d-kin bulkerrd-kiin-tiihhj--mmmmm
0.1
65.13+:3.72
65.24++2.10
14.19+1.22
1..17+1.83
62.10+5..88
70.39+
0.05
78. 1.-22.50
59,21
70.20+2.71
Walker2d-kine sineerdd-kin-thigjjt--aanat
1..50
61.96:1..71
55.95-116.29
1
69.45-11.78
66.92-10.04
71.38:15.42
different tasks and environments. It is interesting to note that even the suboptimal parameters (0.05)
in Vable/ooutperr outperform the baseline algorithms.
Rule of thumb hyperparameters. We notice that there is no universal set of hyperparameters that works well across all tasks with different environments, shift types, and levels. Even without the dynamics shift, model-based RL methods typically require different hyperparameters for different environments. But empirically,
could have set of hyperparameters that generally receives relatively good performance for most tasks, i.c.. weight of BC = 0.1 and MOPO penalty = 5. From there, we primarily tune the BC loss weight based on the convergence behavior of the policy. In most cases, using a MOPO penalty of 5 and BC loss weight selected from the range 0,05, 0.1, 1, 2 yields strong performance. Overall, the number of hyperparameters is modest compared those commonly required in offline model-basee RL methods.
Computational Resources We run all experiments on a single GPU (NVIDIA RTX A5000, 24,564 MiB) paired with 8 CPUs (AMD Ryzen Threadripper 3960X, 24-Core). Each experiment requires approximately GB of RAM and 20 GB of available disk space for data storage.
Computational Cost We provide an estimated running time of MOPO, DARA, BOSA and our method in Appendix C.5 The running time of MOBODY requires approximately 25% more time to run million steps compared to model-free DARA and is faster than the BOSA. The extra running time due to the dynamic learning and generation of rollouts. On the other hand, MOPO and MOBODY have similar running times. This demonstrates that we have similar computational cost and running time compared to the existing model- based method, as the additional loss calculation doesn't 't significantly increase the computation time.
22

---

Published as a conference paper at ICLR 2026
Table 10: Running time comparison on A5000, AMD Ryzen Threadripper 3960X 24-Core Processor.
Grrviy2d
HaliCheetab--raviit--S3
BOSA
~3 hours
~3.5 hours
DARA
~2 hours
2.5 hours
MOPO
2.5 hours
3 hours
MOBODY
2.5 hours
3 hours
C.6 ENVIRONMENT SETTING
Gravity Shift. Following the ODRL benchmark (Lyu et al., 2024b), we modify the gravity of the environment by editing the gravity attribute. For example, the gravity of the HalfCheetah in the target modified to 0.5 times the gravity in the source domain with the following code.
gravity <opt ion gravity-"O 0 -4.9009 905" "itestep="O.0110
Friction Shift The friction shift is generated by modifying the friction attribute in the geom elements. The frictional components are adjusted {0.1, 2.0, 5.0) times the frictional components in the source domain, respectively.
Kinematic Shift The kinematics shift simulated through broken joints by limiting the rotation ranges of some hand joints. We consider the broken ankle joint, hip joint, foot joint, etc, for Mujoco and Adroit environments.
Morphology Shift The morphology shift is achieved by modifying the size of specific limbs or torsos of the simulated robot Mujoco and shrink the finger size in the manipulation task, without altering the state space and action space.
D LIMITATION AND FUTURE WORK
MOBODY relies on the assumption that the source and target domains share common state representation óg and transition ér that map the unified latent state action representation to the next state. We believe this assumption is reasonable in our setting: the source and target domains share underlying structure, and MOBODY is designed to exploit this while allowing domain-specific differences via separate action encoders for the two domains. Empirically, we evaluate MOBODY under various types and levels of dynamics shift, and observe that it outperforms or matches recent ODRL baselines across almost all benchmark settings. In particular, in large-shift cases where existing methods fail, MOBODY performs significantly better than the baselines. This suggests that our shared- maapedg assumption not overly restrictive and remains applicable in many off-dynamies RL scenarios.
In extreme cases where the shift large that this shared-structure assumption breaks down, the key assumptions of existing baselines (e.g., DARA's low-shift-region assumption) would also fail. In such regimes, more advanced techniques such as stronger domain adaptation or zero-shot transfer may be needed, which we see as interesting direction for future work.
Also, similar baselines, our method also struggles with the sparse-reward settings like Antmaze. We believe that tackling sparse-reward off-dynamics RI. is an important and challenging future research direction that requires substantially different methods from those the current literature.
USAGE OF LLM
All ideas and research are conducted by the author, and the paper itself is written by the author. The LLM is used as tool for polishing the written content of the paper and checking the grammar errors.
23

---

Published as a conference paper at ICLR 2026
Table 11: Hyperparameter the MOBODY and baselines.
Hyperparameter
Value
Shared
Actor network
(256, 256)
Critic network
(256, 256)
Learning rate
x 10
Optimizer
Adam
Discount factor
0.99
Replay buffer size
106
Nonlinearity
ReLU
Target update rate
x 10
Source domain Batch size
128
Target domain Batch size
128
MOBODY
Latent dimensions
State encoder
(256, 256) (32) (256, 256)
State action encoder
Transition
Representation penalty Axep
Rollout length
1, or 3
MOPO-Style Reward Penalty 3
1,5 or 10
Q-weighted behavior cloning
0.05, 0.1 1
Classifier Network
(256, 256)
Reward penalty coefficient
0.1
DARA
Temperature coefficient
0.2
Maximum log std
Minimum log std
-20
Classifier Network
(256, 256)
Reward penalty coefficient
0.1
BOSA
Temperature coefficient
0.2
Maximum log std
Minimum log std
-20
Policy regularization coefficient Apolicy
0.1
Transition coefficient Auansition
0.1
Threshold parameter e,
log(0.01)
Value weight w
0.1 for the behavior policy, for the dynamics model
CVAE ensemble size
IQL
Temperature coefficient
0.2
Maximum log std
Minimum log std
-20
Inverse temperature parameter /
3.0
Expectile parameter
0.7
TD3_Be
Normalization coefficient
BC regularization loss
0.05, 0.1 or
MOPO
Transition
(256,256,256)
Maximum log std
Minimum log std
-20
Reward penalty r
1, or 10
Rollout Length
1, or
24


Reviews:
[{"review_id": "qhi7Q3JMgW", "reviewer": "Reviewer_WLw7", "summary": "The paper proposes MOBODY, a model-based approach to off-dynamics offline reinforcement learning, addressing the setting where the source and target environments share rewards but differ in dynamics. Unlike prior methods that penalize or exclude high-shift samples, MOBODY leverages both limited target data and abundant source data through separate action encoders and a shared latent transition model. A cycle-transition loss is introduced to stabilize representation learning, while a target-Q-weighted behavior cloning loss guides the policy toward high-value target actions. Experimental results on MuJoCo and Adroit benchmarks show that MOBODY consistently outperforms existing off-dynamics and model-based baselines, particularly under large dynamics shifts.", "questions": "- Could the authors provide any theoretical guarantees (e.g., convergence, stability, or error bounds) for MOBODY, even under simplified assumptions? How does the model behave under known pathological dynamics mismatches?\n- In settings where MOBODY's improvements over baselines are small (e.g., Walker2d-Friction-2.0), or where it underperforms, what factors contribute to this variance? Could the authors provide an analysis or hypothesis for these outcomes?\n- How does MOBODY perform in sparse-reward environments, where exploration is more challenging and informative transitions are rare? Could the authors include results or discussion in such settings?\n- How sensitive is MOBODY to the quantity of target data? Have the authors evaluated performance when fewer than 5,000 transitions are available?\n- Several return-conditioned or return-based algorithms (e.g., Decision Transformers, RADT, or return-augmented models) have shown promising results in off-dynamics offline RL. How does MOBODY compare conceptually and empirically to such return-based approaches?\n\n## Reference\n- **[1]** Miyaguchi, Kohei. \"Worst-case offline reinforcement learning with arbitrary data support.\" Advances in Neural Information Processing Systems 37 (2024): 61481-61512.\n\n- **[2]** Brandfonbrener, David, et al. \"When does return-conditioned supervised learning work for offline reinforcement learning?.\" Advances in Neural Information Processing Systems 35 (2022): 1542-1553.\n\n- **[3]** Cen, Zhepeng, et al. \"Learning from sparse offline datasets via conservative density estimation.\" arXiv preprint arXiv:2401.08819 (2024).\n\n- **[4]** Rengarajan, Desik, et al. \"Reinforcement learning with sparse rewards using guidance from offline demonstration.\" arXiv preprint arXiv:2202.04628 (2022).\n\n- **[5]** Wang, Ruhan, et al. \"Return augmented decision transformer for off-dynamics reinforcement learning.\" arXiv preprint arXiv:2410.23450 (2024).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a clear and well-motivated model-based framework for off-dynamics offline reinforcement learning. Unlike prior work that relies on reward regularization or data filtering, it directly learns the target dynamics, offering a fresh and practical way to handle dynamics mismatch in offline settings.\n- The paper introduces MOBODY, which shows solid and consistent improvements over existing baselines on several off-dynamics offline RL benchmarks.\n- This paper provides a comprehensive empirical evaluation across both MuJoCo and Adroit environments, covering multiple types of dynamics shifts—gravity, friction, kinematics, and morphology—at varying levels of severity.\n- The paper is well written, with clear organization and logical flow throughout. In particular, the results are effectively visualized—figures and tables are well designed, easy to interpret, and enhance the overall clarity and impact of the presentation.", "weaknesses": "- Lack of theoretical analysis supporting the proposed approach. The paper does not provide any convergence guarantees, error bounds, or stability analysis, which are important for understanding the conditions under which the algorithm is expected to perform reliably [1-2]. Given the complexity of the model and the presence of domain shifts, some theoretical insights—even in simplified settings—would significantly enhance the rigor and credibility of the work.\n- While MOBODY generally outperforms existing baselines, some of the reported improvements appear marginal or potentially within the range of statistical noise—for example, in the Walker2d-Friction-2.0 setting, where the gains over baselines are minimal. Moreover, in a few scenarios, MOBODY underperforms relative to other methods. The paper would benefit from a more detailed analysis or discussion of these cases to help contextualize performance variance and identify potential limitations of the approach.\n- All experiments are conducted in dense-reward environments, which may not fully capture the challenges faced in sparse-reward settings. It would strengthen the paper to include an analysis of MOBODY's performance in sparse environments, where exploration and generalization are typically more difficult [3-4].\n- Although the method is described as operating with \"limited target data,\" it still assumes access to 5,000 transitions, which may not reflect truly scarce or low-resource settings. It remains unclear how MOBODY would perform when target data is extremely limited, noisy, or when the dynamics mismatch is highly non-smooth or discontinuous. Evaluating the sensitivity of the model under such challenging conditions would help clarify the robustness and practical applicability of the approach.\n- The proposed framework introduces multiple components—such as separate action encoders for the source and target domains (ψ_src, ψ_trg) and a VAE-based cycle transition module—which likely increase both training time and memory usage. However, the paper does not provide any runtime, training cost, or sample-efficiency comparisons against baseline methods, making it difficult to assess the practical overhead of the approach.\n- While the paper compares MOBODY against several strong baselines, it overlooks a class of return-conditioned or return-based methods that have also demonstrated competitive performance in off-dynamics offline RL settings [5]. Including representative algorithms from this category—such as decision transformer variants or return-augmented models—would provide a more comprehensive evaluation and help clarify whether the proposed approach offers advantages beyond standard model-based and reward-regularized baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C0jrVbUAMK", "reviewer": "Reviewer_NauK", "summary": "This paper tackles the \"off-dynamics offline reinforcement learning\" (ODRL) problem, where the goal is to learn a target-domain policy using a large source-domain dataset and a very small target-domain dataset, which have different dynamics. The authors propose MOBODY (Model-Based Off-Dynamics Offline RL), an algorithm that learns a model of the target dynamics to generate synthetic data, allowing the policy to explore high-reward regions that may be in high-shift areas, which existing methods often fail to do.", "questions": "1. Could you elaborate on why methods that directly regularize the policy's state-visitation distribution (like SRPO) were not considered as baselines, given that they directly address the problem of distribution shift under new dynamics?\n\n2. The Q-weighted BC loss (Eq. 8) is very similar to IQL's advantage-weighted regression. What is the performance of your method if you simply replace your loss with the IQL policy loss? This would clarify whether the specific formulation of Eq. 8 is a key part of your contribution.\n\n3. How much of the performance gain comes from the model-based rollouts versus the superior dynamics-learning architecture? (i.e., what is the performance of MOPO if it uses your dynamics model?)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper correctly identifies a key weakness of current ODRL methods (e.g., DARA, BOSA): their conservatism, which stems from data filtering or reward penalization, prevents them from exploring high-reward states that may exist outside of low-dynamics-shift regions.\n2. The central idea of using separate action encoders while sharing state and transition functions is a clever and intuitive way to leverage the large source dataset to learn shared structural knowledge while still capturing the domain-specific differences with the small target dataset.", "weaknesses": "1. The paper fails to cite or compare against a relevant class of transfer learning and offline RL methods. The related work discusses reward regularization (DARA) and data filtering (VGDF), but it omles a large body of work on policy regularization, specifically methods that use state-distribution matching or regularization. For example, State Regularized Policy Optimization (SRPO) and similar approaches that explicitly constrain the policy's state-visitation frequency to be close to a trusted distribution (like the target dataset) are highly relevant. Such methods provide an alternative and potentially more direct way to handle the dynamics shift than the complex model-based approach proposed here.\n2. While the dynamics learning architecture is interesting, the policy learning side feels incremental and like a complex combination of existing ideas.\n- The \"enhanced target data\" buffer is a simple-enough combination of target data, model rollouts (a standard model-based RL technique, e.g., MOPO), and augmented source data (using the existing DARA method).\n- The \"target Q-weighted behavior cloning loss\" (Eq. 8) is presented as a novel contribution but is a minor variant of advantage-weighted regression, which the paper itself cites as an inspiration (IQL). The paper provides no theoretical or strong empirical justification (e.g., a direct comparison) for why its specific formulation ($\\exp (Q / \\operatorname{avg}|Q|) \\times(\\pi(s)-a)^2$) is superior to the more established and better-understood formulations used in methods like IQL.\n3. The final MOBODY algorithm is highly complex. It stitches together a VAE-style dynamics learner with an ensemble for uncertainty, the full DARA framework for reward augmentation, and a custom Q-weighted policy optimizer. This high number of \"moving parts\" makes the method difficult to reproduce, tune, and analyze, and it obscures which components are providing the most benefit.\n4. The motivation for the Q-weighted loss is purely intuitive. There is no analysis of its properties. Why use an exponential weight? Why normalize by the average absolute Q-value? How sensitive is this to the Q-value estimates, which are known to be noisy? A more principled derivation or analysis is needed to justify this specific design over simpler, existing alternatives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JJNtIsL0tA", "reviewer": "Reviewer_MfFn", "summary": "**Summary.** This work studies how to tackle the problem of off-dynamics offline RL. More precisely, off-dynamics offline RL aims to learn a policy using offline data from a source domain and a much smaller amount of data from a target domain. Main challenge limits exploration and adaptation to the target's optimal trajectories when the dynamics shift is significant. To alleviate this issue, this paper introduces MOBODY that explicitly learns an unified state representation and separate action encoders for source and target domains. They further introduce a Q-weighted behavior cloning technique to bias the policy toward high-value actions instead of naive action imitation. The method achieves consistent improvements over strong baselines on MuJoCo and Adroit benchmarks with different types and severities of dynamics shift.\n\n---\n**Review summary.** Overall, the reviewer thinks that the paper is clearly written, well-motivated, and timely, especially in the context of sim-to-real transfer. The idea itself is conceptually simple, intuitively reasonable, and appears to be practical to implement. Although the work is not theoretically deep, it is coherent and sufficiently justified as a framework-oriented contribution. However, the experiments section leaves several open questions and limitations that need further clarification. Therefore, the reviewer assigns an initial score of 6 and plan to revisit this rating after the authors address the concerns and questions raised in this review.", "questions": "- In Table 1, IQL often outperforms specialized methods such as DARA and BOSA. Could the authors explain why this occurs?\n    - Have the authors tried using AWR style policy extraction for DARA or BOSA to see whether the observed gap stems from the extraction mechanism rather than the dynamics modeling?\n- For Ablation 1, would it be possible to combine alternative dynamics models, like COMBO or RAMBO, with the proposed Q-weighted BC to isolate the contribution of the policy regularizer?\n- For Ablation 2, could the authors replace the Q-weighted cloning loss with an AWR to directly compare different policy extraction approaches?\n- BOSA appears conceptually compatible with MOBODY’s dynamics learning. Have the authors tested whether integrating the two (BOSA + MOBODY dynamics) yields complementary effects or insights into the efficacy of Q-weighted behavioral cloning?\n- Given the extremely small target dataset, have the authors observed any instability or collapse in the target encoder $\\psi_{\\text{trg}}$?\n- How does MOBODY perform when the amount of target-domain data decreases further, e.g., below 1 K samples?\n- How many ensemble models are used for uncertainty quantification? \n    - Is the performance sensitive to ensemble size or the uncertainty penalty $\\beta$?\n- The reviewer wonders what the aurhos predict will happen if we consider a representation based on the TD information of each domain.\n    - A. Zhang, et al. Learning Invariant Representations for Reinforcement Learning without Reconstruction. ICLR 2021.\n    - C. Zheng, et al. Contrastive Difference Predictive Coding. ICLR 2024.\n    - D. Lee, et al. Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning. ICML 2025.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Writing**\n- The paper is very readable and logical from motivation to formulation to algorithm and experiments.\n- Figures and tables are generally informative; the method and notation are consistent.\n\n---\n**Methodology**\n- The central idea, learning shared latent state and transition functions with domain-specific action encoders, is simple and well-motivated.\n- The architecture (shared $\\phi_E$, $\\phi_T$; separate $\\psi_{\\text{src}}$, $\\psi_{\\text{trg}}$) directly targets the off-dynamics challenge, that is, achieving the same actions but different dynamic transtion.\n- The introduction of a Q-weighted BC is reasonable, encouraging the policy to favor actions with high target-domain value while mitigating overfitting to source data.\n\n---\n**Experiments**\n- The experiments are extensive, covering MuJoCo with diverse transition (e.g., gravity/friction/kinematic/morphology shifts) and Adroit tasks with multiple difficulty levels.\n- Ablation studies demonstrate the impact of both the cycle transition loss and Q-weighting.\n- Computational cost, hyperparameter sensitivity, and reproducibility are reported in detail.", "weaknesses": "**Writing**\n- Figures 3–4 summarize results but are under-analyzed. differences and error ranges are not interpreted in the text. Please discuss more why and how come.\n- Algorithm 1 and 2 include many moving parts, such as DARA regularization and reward modeling, that are not clearly integrated into the paper.\n \n**Methodology**\n- It is not fully clear how the encoder losses (Eq. 4) interact within the overall objective (Eq. 6). Are $L^{\\text{src}}{\\text{rep}}$ and $L^{\\text{trg}}{\\text{rep}}$ averaged or weighted differently?\n- The paper should clarify whether $p(s'|s,a)$ is assumed deterministic. This strongly affects the validity of the VAE-style cycle transition loss and the interpretation of uncertainty.\n- The cycle transition loss (Eq. 5) is creative, but its formal grounding is weak: the gradient-stopping mechanism and KL structure are under-explained, and it is unclear how it avoids mode collapse or degenerate representations.\n- The target-domain encoder $\\psi_{\\text{trg}}$ is trained with very limited data. Overfitting and instability risks should be analyzed or mitigated via regularization or weight sharing.\n- Target Q-weighted BC is empirically validated but lacks theoretical analysis; no formal bound or convergence argument is provided, and the technique is a relatively mild variation of AWR.\n\n**Experiments**\n- The reviewer concerns baseline selection and fairness.\n    - Most baselines (IQL, TD3-BC, DARA, BOSA, MOPO) are strong but somewhat outdated.\n    - More recent or structurally related methods (e.g., BPRL, LEO, Koopman Q-learning, or model-based domain adaptation approaches) are missing.\n- In several tasks, simpler methods, IQL, outperform off-dynamics-specific methods, i.e., DARA, BOSA. This discrepancy deserves explicit discussion.\n- Only two dynamics domains are considered. It is unclear how MOBODY generalizes to multiple or continuous domains.\n- High variance in results, for example, Walker2d-Gravity 5.0: $46.05 \\pm 20.73$, suggests instability that is not analyzed.\n- There is no qualitative evidence supports cross-domain transfer.\n\n**Limitations**\n- MOBODY assumes that a shared transition mapping ($\\phi_E$, $\\phi_T$) exists across domains. This is a strong assumption; when dynamics are structurally divergent, the approach may fail.\n- The approach relies on uncertainty regularization from model ensembles, but this is treated superficially. \n- There is no analysis of behavior under extreme dynamics mismatch, where shared representations might be misleading.\n- Need to more discuss and survey about prior connections.\n\n**Miscellaneous**\n- Eq. (3): inconsistent summation notation and formatting.\n- Eq. (6): unclear aggregation of $L_{\\text{rep}}$ terms.\n- Line 165: justification for adding $z_s$ to $\\psi(z_s,a)$ is weak (“for simplicity” is insufficient).\n- Line 126–127: references needed for conservative reward regularization (e.g., MOPO, COMBO).\n- In several places, “off-dynamics RL” should be replaced with “off-dynamics offline RL” for accuracy.\n\n**References**\n- M. Weissenbacher et al. Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics. ICML 2022.\n- J. Eo, et al. The impact of dataset on offline reinforcement learning performance in UAV-based emergency network recovery tasks. IEEE Comm. Let. 2023.\n- M. Uehara, et al. Pessimistic model-based offline reinforcement learning under partial coverage. ICLR 2022.\n- B. Hao, et al. Bootstrapping fitted q-evaluation for off-policy inference. ICML 2021.\n- G. An, et al. Uncertainty-based offline reinforcement learning with diversified q-ensemble. NeurIPS 2021.\n- D. Lee, et al. Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning. ICML 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)