ADM-v2: Pursuing Full-Horizon Roll-out in Dynamics Models for Offline Policy Learning and Evaluation
Authors: Haoxin Lin
Siyuan Xiao
Yi-Chen Li
Zhilong Zhang
Yihao Sun
Chengxing Jia
Yang Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Model-based Reinforcement Learning
Offline Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We propose ADM-v2, a dynamics model that enables accurate and efficient full-horizon rollouts, achieving new SOTA on both off-policy evaluation and offline RL benchmarks.

Abstract: Model-based methods for offline Reinforcement Learning transfer extensive policy exploration and evaluation to data-driven dynamics models, effectively saving real-world samples in the offline setting. We expect the dynamics model to allow the policy to roll out full-horizon episodes, which is crucial for ensuring sufficient exploration and reliable evaluation. However, many previous dynamics models exhibit limited capability in long-horizon prediction. This work follows the paradigm of the Any-step Dynamics Model (ADM) that improves future predictions by reducing bootstrapping prediction to direct prediction. We structurally decouple each recurrent forward of the RNN cell from the backtracked state and propose the second version of ADM (ADM-v2), making the direct prediction more flexible. ADM-v2 not only enhances the accuracy of direct predictions for making full-horizon roll-outs but also supports parallel estimation of the any-step prediction uncertainty to improve efficiency. The results on DOPE validate the reliability of ADM-v2 for policy evaluation. Moreover, via full-horizon roll-out, ADM-v2 for policy optimization enables substantial advancements, whereas other dynamics models degrade due to long-horizon error accumulation. We are the first to achieve SOTA under the full-horizon roll-out setting on both D4RL and NeoRL. The code is available at https://github.com/LAMDA-RL/adm2.

Published as a conference paper at ICLR 2026
ADM-v2: PURSUING FULL-HORIZON ROLL-OUT IN DYNAMICS MODELS FOR OFFLINE POLICY LEARNING AND EVALUATION
Haoxin Lin', Siyuan Xiao', Yi-Chen Li', Zhilong Zhang', Yihao Sun', Chengxing Jia', Yang Yu
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
& School of Artificial Intelligence, Nanjing University, Nanjing, China
University of Montréal, Montréal, Canada
(linhx, xiaosy, liye, zhangz1] ) @lamda in ju. edu. on
yihao. sun@mi l quebec, {jiacx, yuy ) 0 1 lamda. n ju. edu. en
ABSTRACT
Model-based methods for offline Reinforcement Learning transfer extensive policy exploration and evaluation to data-driven dynamics models, effectively saving real-world samples in the offline setting. We expect the dynamics model to allow the policy to roll out full-horizon episodes, which is crucial for ensuring sufficient exploration and reliable evaluation. However, many previous dynamics models ex- hibit limited capability in long-horizon prediction. This work follows the paradigm of the Any-step Dynamics Model (ADM) that improves future predictions by reducing bootstrapping prediction direct prediction. We structurally decouple each recurrent forward of the RNN cell from the backtracked state and propose the second version of ADM (ADM-v2), making the direct prediction more flexible. ADM-v2 not only enhances the accuracy of direct predictions for making full- horizon roll-outs but also supports parallel estimation of the any-step prediction uncertainty to improve efficiency. The results on DOPE validate the reliability of ADM-v2 for policy evaluation. Moreover, via full-horizon roll-out, ADM-v2 for policy optimization enables substantial advancements, whereas other dynamics models degrade due to long-horizon error accumulation. We are the first to achieve SOTA under the full-horizon roll-out setting on both D4RL and NeoRL.. The code is available at
INTRODUCTION
Offline Reinforcement Learning (Fujimoto Gu 2021 Kumar et al., 2020 a promising way to solve real-world decision-makee tasks. The core challenge lies in learning well well-performing policy from a fixed dataset and conducting off-policy evaluation the offline training process. Model-based Reinforcement Learning (MBRL) (Yu et al.) 2020 202T [unn et al. 2023 Lin et al. 2025) in the offline setting is well-suited to addressing the challenge of limited data, since many policy exploration and evaluation can happen within dynamics models. The ideal case is that the dynamics model can roll out complete episodes, acting as a simulator that closely resembles the real world.
An important role of full-horizon roll-outs is enable the policy to sufficiently explore, which is beneficial for achieving better performance. (Sims et al. (2024) finds that the truncation of roll-outs hinders the policy from reaching some edge states, thereby impacting the estimation of the value function. Exploration of edge states facilitates more thorough policy learning and yields further improvements. Additionally, [u et al. (2022) finds that longer roll-outs combined with suitable uncertainty penalty lead to better performance. On the other hand, offline policy evaluation (Fu et al. 202T, Yang et al. 2020 Chen et al. 2024b) also imposes requirements on the model's 's ability make long- Long-horizon roll-outs. With data-driven dynamics model, the most natural approach for policy evaluation to sample several complete trajectories in the model and estimate the expected return.
"Corresponding Author

---

Published as a conference paper at ICLR 2026
Although many prevalent MBRL algorithms et 2020 Sun et al. 2023) are theoretically derived under the assumption of full-horizon model roll-outs, practically they adopt short and branched roll-outs (Janner al. prediction, which attributes the next state to the prediction of the current state and causes model error accumulation (Xu et al. (02T)) during roll-outs. This limitation is commonly faced in almost all
2019p. This is because their dynamics models rely the bootstrapping
popular dynamics models, including ensemble models (Chua et al., 20188] Yu et al.] 2020), causal models (Zhu et al., 2022). and adversarial models (Chen et al.) [023] Bhardwaj et al. (023). For
them, roll-out truncation IS compromise to ensure the reliability of model- -eneratted samples.
In this paper, we seek to extend the roll-out length to match the episode horizon and follow the Any- step Dynamics Model (ADM) (Lin et al. (025). ADM directly models the state transitions resulting from executing any-step actions through a GRU cell (Cho al. 2014), showing the effectiveness of improving future predictions by reducing bootstrapping prediction to direct prediction. ADM generalizes the direct state transition after executing a multi-step action sequence (Asadi et al.) 2018)
2019; Che et al. 2018 Machado et al., 2023) into any-step formulation. Specifically, ADM takes as imput a starting state 5y and an action sequence of arbitrary length k. (ag, (dp:a1," ... ak-1), to
predict Sμ. To match the length of the action sequence, ADM duplicates the starting state multiple times. Therefore, the actual input to the GRU inside ADM is a state-action sequence of the form ([5o; 0g]: [8, an),.... [so; ak-1]). I]), making the network architecture appear inflexible.
The structure of the original ADM not only induces strong coupling between the GRU's hidden representations and the starting state but also prevents any-step prediction from being accelerated through parallel computation. To address these issues, we propose the second version of ADM (ADM-v2). ADM-v2 encodes the starting state so into a latent vector hig which is directly used as the initial hidden state of the GRU. The information of 9g is contained within hyg so each forward of the GRU no longer takes Sy as input but only the action. ADM-v2's decoupling of sy reduces the impact of perturbations in so on multi-step direct prediction, thereby improving the prediction reliability. Moreover, ADM-v2 discards the backtracking mechanism (Lin et al. 2025) proposed in
ADM to obtain any-step prediction. We design a roll-out algorithm for ADM-v2 to enable parallel computation of direct predictions from different strides, called Parallel Any-step Roll-out (PARoll). PARoll also allows the parallel estimation for any-step uncertainty (Lin et al. 2025), which is efficient for policy learning and evaluation with full-horizon roll-outs in offline settings.
In general, our contributions are summarized as follows. (1) We propose the ADM-v2 and the roll-out algorithm PAlll.. The experiments on D4RL (Fu et al. 2020) dataset demonstrate the advantages of ADM-v2 in terms of prediction accuracy and efficiency, especially during full-horizon roll-outs. (2) We evaluate the given policy offline by rolling out complete episodes in ADM-v2. Results on DOPE (Fu et al. 2021) benchmark show that this approach yields more accurate policy evaluation than other off-policy evaluation methods. (3) We use the full-horizon roll-outs sampled in ADM-v2 for policy learning and apply any-step uncertainty as penalty term in Q-value estimation. This algorithm, ADM2PO-fh, achieves SOTA performance on both the D4RL and NEORL (Qin et al. 2022) datasets.
PRELIMINARIES
2.1 MARKOV DECISION PROCESS
Markov Decision Process (MDP) is defined by the tuple M (S, Po.7). Here, S is the state space, A is the action space, and ((s441: at) denotes the probability density function of the next state 8:+1 and the reward r++1 given state 8; and action a, Po is the initial state distribution over S. 7 E [0, 1) is the discount factor. We use p² to denote the on-policy distribution over states induced by the dynamics function T and the policy म. The objective of RL is to learn an optimal rr...
policy x² that maximizes the expected discounted return Ep- [E
2.2 OFFLINE MBRL
In the offline setting, the agent only has access to fixed dataset of transitions Deny ((s,, 0e, Tit1, 811)), collected by the behavior policy in the real world. In the offline MBRL framework, dynamics model T is learned from Dan by maximizing the expected likelihood E ret Dem. [log (( 1, 1,t+1,7t-11 (r-1/11,))) This allows us construct surrogate MDP
2

---

Published as a conference paper at ICLR 2026
Skito
GRU
Towudiio
Decoder
Decocer
Dooode
Decocer
ADME Uni
ADM2 Un!
fre
Seam foom
ADM
(b) ADM-2
Figure I: Illustration of (a) the vanilla Any-step Dynamics Model (Lin et al. 2025); ) (b) the second
version of Any-step Dynamics Model (ADM-v2), both are structured using a GRU Cell.
M = AS, Po. 7). Both the offline dataset Dan and the rolled-out data by the learned model T, denoted as Dmodel. are utilized to optimize the policy. To mitigate the issue of model coverage, previous works (Yu et al. 2020; Kidambi et al., (2000) incorporate the ensemble-base based uncertainty as a penalty into the reward function, preventing the agent from sampling within unsafe regions of Í.
2.3 ANY-STEP DYNAMICS MODEI.
Lin al. (2025) proposes the Any-step Dynamics Model (ADM) T( Tt+&/ axa+k-1) to model the distribution of St-k and r*+ after executing a k-step action sequence Gc++k-1 = (oe, Ot-::". a+++-1) 8t, where k be any integer under the limit of the maximum stride m. For future state prediction at each step, ADM has m backtracking choices. For instance, (st, ac), (st-1, at-1:) (;-2:));; (s Or-m+1:t) are all selectable as ADM's inputs to predict St+1- After reducing bootstrapping prediction to direct prediction, ADM can mitigate the compounding error model roll-outs. What's more, the naturally diverse predictions from different backtracking choices can be utilized to estimate the model uncertainty, which has shown great consistency with the model error ADMPO (Lin al. 2025) and Whale (Zhang et al. 2024b).
METHOD
In this section, we design the second version of ADM (ADM-v2) support full-horizon roll-outs and propose an efficient algorithm called PARoll to speed up any-step roll-out with uncertainty estimation. Furthermore, we introduce how to evaluate and learn the policy offline with ADM-v2 and PARoll.
3.1
ADM-V2
The architecture of ADM-v2 is clear and straightforward, as illustrated in Figure lb). We also illustrate ADM in Figure I aa) for comparison. Following the definition of ADM (Lin et al. 2025).
ADM-v2 can be denoted by To ( T (rt4k(11))))) 1). where 0 represents the neural parameters. k is the stride of the multi-step direct prediction, which be any integer limited by the maximum prediction stride m, i.e., < k < m. The input of ADM-v2 is state 8t and a k-step action sequence a1:44-11. The output modeled two independent Gaussian distributions of s1+.k and Tr+h-
To handle the input action sequence with variable step sizes, we utilize an RNN Elman 1990) with a GRU (Cho al., 2014) to implement ADM-v2, the same as ADM. The difference is that ADM-v2 decouples each recurrent forward of the GRU cell from the backtracked state Then the GRU cell inside ADM-v2 no longer takes S; as input repeatedly. There are three elements within ADM-v2. including a GRU go, state encoder ence, and transition decoder decs. Both the state encoder and transition decoder are implemented using MLPs [Goodfellow et al. 2016). As a whole, go. ence, and decş constitute Ta together.
The backtracked state is encoded to latent vector ht by the state encoder, i.e., ho ence(st) We directly view he as the initial hidden state of the GRU cell. With h and a; as input, the hidden po(h,,aa). Next, the transition decoder decodes
state is updated to h4t1 by GRU, i.e., h411 h to the distributions of 8441 and re+1- Concretely, (HC+ 25.1.%% E = (h,-1). then N ((1441 Fit)) and N (47411 ) becomes the Gaussian distributions for st+1 and re+1
3

---

Published as a conference paper at ICLR 2026
Store Inceder
ADM2 List
ADME Lizn
ADME Line
ADME
Use
ATM
ADM-x
ATMM-2
State Exceder
ADM
ADME
ATME
ADME
AllM-22
AIMM-2
ADM
Roll-out Init
fa--3
Esoada
Stare
ADM?
ADM-22
Figure 2: Illustration of the Parallel Any-step Roll-out (PARoll), which designed based on ADM-v2 and able provide diverse predictions in parallel.
respectively. We regard the combination of GRU and transition decoder as an integrated ADM2 Unit. The hidden state h4+1 can be continually propagated to the ADM2 unit to predict subsequent states and rewards, until the maximum stride m reached.
As an end-to-end end neural network, ADM-v2 does not rely on any explicit supervision for intermediate variables, including the hidden state h. The joint distribution for each stride (1 S S m) is optimized by directly maximizing the log likelihood
(0)
log (S To+k aer+h-1))
(I)
which is averaged among m strides. The gradient of Equation equation Tis directly backpropagated through the GRU, state encoder, and transition decoder.
ADM-v2 can improve future predictions by reducing bootstrapping prediction multi-step direct prediction. For long-horizon prediction, we divide roll-outs into multiple windows of length m. The first window starts from the real state 8n, with the following m states predicted without any error accumulation, and ends at the predicted 8m. For the n-th window (n >> 1), the starting state 6(n-1)m encoded by the state encoder to hnn-1)m- Along with the implicit update from ft 1)m to fyom through GRU forward propagations, ((m11)m+1 8umm are predicted without state bootstrapping. Therefore, the state bootstrapping only occurs when switching between windows. The roll-out process described above does not account for prediction diversity and uncertainty estimation. In the following subsection, we introduce the Parallel Ate-step Roll-out (PARoll) to address these limitations.
3.2 PARALLEL ANY-STEP ROLL-OUT
As introduced by ADM (Lin et al. 2025), one advantage of ADM is to generate diverse predictions and estimate the model uncertainty without an ensemble. However, ADM has to backtrack historical states from different distances and recursively call the GRU cell at each step, which costs unnecessary time consumption during roll-outs. The details are described Appendix
ADM-v2 removes the mechanism of backtracking and rolls out diverse predictions in parallel, as shown by Figure 2 the illustration PAR At the beginning of a roll-out, we sample batch of state-action sequences of m steps from Denv. (8o, 4g: are m selectable states at different time steps, each corresponding to a timeline for the roll-out. For the first timeline, after encoding Sp to hn and recursively feeding (4g- a.,-.. am 2) into ADM2
40 81, A1,''* 8mm-2, 4m- 8m-1). There
Unit, we obtain the first hidden state at step m - 1, denoted as h (1) where the superscript indicates the index of the timeline and the subscript indicates the time step. Generally, for the i-th (1 < i < m)
timeline, after encoding 1 to h and recursively feeding (a:-1, 0;, ... am-2) into ADM2 Unit,
obtain the i-th hidden state (1) at step m 1. Specially, the m-th hidden state h (m)) is directly obtained after encoding the 8m-1. After the process of roll-out initialization above, we obtain m diverse starting hidden states (1) mm (1 . ((m))
step m for ADM-v2.
4

---

Published as a conference paper at ICLR 2026
At the first step of the roll-out, the agent chooses action am-1 according to am-1 TT (-11smm-) Then m hidden states are in parallel updated by the ADM2 Unit after inputting am-1- For each i 21,2, m.. after updating the hidden state by hi go go am-1). we have
(1) E Pr I decs (h!)), s Nip
PM N 1
(2)
(om) 9m
)/m)) are m predictions for the state 8m at time step m, transitioned from different backtracked states. Concretely, 8m is transitioned from so after (ag, d1. am-1), Sm is transi-
tioned from 81 after (a), a2,-.. am--1), ...... and g/T² is transitioned from 5m-1 after am-1- The discrepancy in transition strides yields diversity in predictions naturally.
and generating states by decoding h, to s! for each i at each step t m. Both the hidden state updating and decoding operations are finished in parallel. Limited by the maximum stride m set at the training phase of ADM-v2, the hidden state can't be propagated directly along the roll-out. We reset the hidden state when the direct prediction reaches the maximum transition stride for each timeline. For the i-th timeline, the starting state the roll-out initialization is 8;-1, then reset time nodes between direct predictions are (m + 1, 2m + i 3m + i 1, 1....). ). We reset hidden states at corresponding time nodes for each timeline by calling the state encoder to encode predicted states, i.e., ham +i enco ( (s (i)) for the n-th stride. Practically, at each step t, only one timeline needs to reset the hidden state, the timeline (1 mod m) + 2. Overall, each step only needs one state encoder forward. one parallel GRU cell forward, and one parallel transition decoder forward. The pseudo code of PARoll is presented in Appendix D.2 Each roll-out in ADM-v2 is full-horizon.
For further roll-out, we repeat updating hidden states by n, go
For each m 1. we uniformly choose one from the diverse predictions (3
S41 (m)) as
(2)
the prediction and utilize them to estimate the model uncertainty (Yu et al. [020 Kidambi et al. 2020 Lu et al.] [022] Lin et al.) 2025) time step f. For any maximum stride m and the corresponding learned To. the uncertainty of To (st, is quantified
AAD22 (8t, a.) = E Var
Σ k=1
(k) (1()))2) (ft))
=
where The = 1 Ex= Mse The uncertainty term 2/ADM2 can be computed in parallel at each step by PARoll. We make further analysis for this any-step uncertainty in Section 3.3
3.3 ADM-v2 FOR OFFLINE POLICY LEARNING
With a learned ADM-v2, offline policy exploration can directly happen in it. Unfortunately, the policy inevitably falls into some risky regions where the dynamics model is uncertain since the given dataset is limited. Model uncertainty estimation (Yu et al. 2020 Kidambi et al.) 2020 [un et al.) 2023 Lin al. 2025) can penalize actions that would lead the policy to regions
covered by the dynamics model. Our uncertainty term 2/ADM2 is estimated naturally during PARoll. Assuming 44 AM2 is an admissible егror estimator (Yu al. 2020) whose numerical value at each state-action pair bound the model eиror, we can find penalty coefficient /3 to make 3 AM22 become a valid g-uncertainty quantifier (Jin al. 202T), as shown by Theorem 3.1
Theorem 3.1. 3 -4 is a valid E-uncertainty quantifier. Specifically,
f= a:) - (F an) s 8 LAAM22 0t).
(4)
where 7. is the proxy operator induced by ADM-v2 to estimate the true Bellman operator T.
Proof. See Appendix C
According to the sub-optimality theorem derived by Jin et al. (202I). any E-uncertainty quantifier as penalty term during estimation (Sutton& & Barto! 2018) can improve the policy be close to
5

---

Published as a conference paper at ICLR 2026
the optimal policy x² with a bounded optimality gap. Penalizing the original Bellman operator with 3 LADM2, we have
az) :- ((x, 7 B -
(5)
Intuitively, for risky regions, larger model uncertainty corresponds to a looser bound on the Bellman error, thereby increasing the likelihood of Q-value overestimation. Penalizing Q estimation according to the magnitude of uncertainty helps prevent inflated Q-values in uncertain regions. This penalty does not interfere with policy learning on in-distribution samples, since their uncertainty is low and the penalty term remains inactive. Hence, the essence here is to identify policy that achieves great performance within reliable state-action regions. The algorithm of policy learning ADM-v2 via full-horizon roll-out is called ADM2PO-fh, with pseudo code shown in Appendix D.4
3.4 ADM-v2 FOR OFF-POLICY EVALUATION
After learning an ADM-v2 Th from fixed dataset, we directly regard it as a surrogate environment. For any given policy, we can roll out several full-horizon trajectories in T. and compute the average return. The pseudo code of off aff-policy evaluation (OPE) in ADM-v2 shown in Appendix [D.3]
There exists gap between the evaluated return and the true return, whose magnitude depends on the accuracy of long-horizon roll-outs in To. We characterize the relationship between this gap and the model error through Theorem 3.2
Theorem 3.2 (Performance Bound). Suppose the divergence of the k-step transition is bounded as
max E Eenn Drv
(P((sm++*)s Drv(DP(vvnn))))))
(|((ss(tt))mnmm S 8%
max 8h. i indicates the multi-step transition for each E (1,2,-.. .. .m). and 8max k1 .m)
distribution induced by T and the policy and P indicates the multi-step transition distribu- tion induced by the true dynamics T and the policy x. The policy divergence bounded as max, (T
<x. Then the return gap is bounded as
27ymaaxmaa
+ 2
Armax
(n( - グT))
C'6mmax. En).
(6)
(1 7) ) 71-
1-7
Proof. See Appendix
Theorem 3.2 states that the true performance of a policy can be bounded within a range of CC Ex) around the performance evaluated by ADM-v2. The value of C'Emaxx Exy determines
the reliability of ADM-v2 for OPE. If setting 1, the bound becomes 2yrmax
(1-7)) + 2yymaxer
(1-7)²
4rmaxt and is equivalent with the performance bound (Jannrr et al. 2019) the single-star dynamics
('1--"") is satisfied, which is shown empirically in Section 4.1
model. Once the condition dmax < 1-7
can assert that the bound of ADM-v2 is tighter than that of a single-step dynamics model.
EXPERIMENTS
Our experiments focus these main questions: (1) How reliable are the trajectories rolled out by ADM-v2? (2) How effective is off-policy evaluation in ADM-v2? (3) How well is the performance after learning the policy in ADM-v2? (4) How well is the uncertainty estimator e 4ADM2? (5) How does the roll-out horizon affect the performance? (6) Does ADM-v2 learn better hidden representations?
4.1 DYNAMICS MODEL EVALUATION
We evaluate ADM-v2 in terms of prediction accuracy and efficiency, in comparison with four baseline dynamics models: (1) ADM (Lin al. 2025): The original version of ADM, see Appendix E. for details; (2) Ensemble Dynamics Model (EDM) (Janner et al., 2019: Yu et al. 2020; Ni et al. 2025):
EDM consists of several single-step dynamics models learned from different initial neural parameters

---

Published as a conference paper at ICLR 2026
direct vs beotstrapping
full-horizon rall-out
roll-out torghot
predicion error
AlMM-2 mistop luss)
10'
0.45
ADM restop
EDM srapers
10"
100000
0.30
d'a -V-50
coll-ott
100
ADM
50000
0.15
10
Foramer
0.00
10
500
1000
60:M KEM
Piinarre
m
roll-out horizon
dynamiks models
(a)
(b)
(c)
Figure 3: Evaluation of ADM-v2 and baselines on hopper-medium-y averaged over seeds. (a) Comparison between direct prediction and bootstrapping prediction; (b) Compounding error (log scale) curves in full-horizon roll-out; (c) Roll-out throughput (number of samples per second).
and uniformly chooses one to make predictions at each step: (3) Vanilla RNN Dynamics Model (RDM): The ablative dynamics model designed to climinate the performance impact of the RNN itself. It inputs the sequence of state-action pairs instead of only the first state of a stride and the action sequence. See Appendix E222 for details; (4) Proprioceptive Dreamer (P-Dreamer): Dreamer (Hafner et al.] 2020; 2021 2023) focuses on the visual RL tasks. We modify the structure to adapt to the proprioceptive RL tasks where the observation is just a vector, see Appendix E.3 for details.
We conduct the evaluation on D4RL (Fu al. 2020) MuJoCo datasets and demonstrate the complete results in Appendices G.T to G.3 Figure 3 show the result on hopper-medium- dataset. We divide the dataset into training set train dynamics models and an evaluation set evaluate models.
The experiment includes three parts. The first part is the comparison between direct prediction and bootstrapping prediction, shown by Figure 3(a) We sample 100 sequences from the evaluation set and calculate the m--ttep direct prediction error of ADM-v2 and ADM, the m-step bootstrapping
prediction error of EDM, and 8* (1-7")) for each m from to Comparing the curve of ADM-v2 1-7
with )1-7" we can find a proper m to satisfy 6max = (1.2, 5* < ('-2""") While
setting m as the proper value, the performance bound of ADM-v2 is tighter than that of single-step dynamics models like EDM. It is also observed that the direct prediction error curves of ADM-v2 and ADM are both below the bootstrapping prediction eтror curve of EDM, indicating the advantage of direct prediction in improving future predictions. Furthermore, the direct prediction error of ADM-v2 is smaller than that of ADM, showing the structural advancement of ADM-v2 over ADM.
The second part is the evaluation of the compounding error in full-length roll-outs, as shown by Figure (3b) We sample 100 trajectories from the evaluation set and execute actions sequentially in each trained model to calculate the average roll-out error at each step. Since the maximum episode length in MuJoCo (Tooorov et al. (012)) tasks is 1000, set the maximum roll-out horizon to 1000 as well. We observe that the roll-out error curve of ADM-v2 is below the curve of other baselines in most cases. Until the episode end, the roll-out error of ADM-v2 still does not tend toward infinity.
The third part is the evaluation of the roll-out throughput, which is calculated as the average number of roll-out samples per second, measuring the time efficiency of the model roll-out. For fairness, the roll-out batch size of each dynamics model is set to be the same. As shown by Figure 3 the roll-out throughput of ADM-v2 is greater than other RNN-based dynamics models, ADM, RDM, and P-Dreamer. EDM achieves high time efficiency in roll-out due to its simple architecture. However, the disadvantage of ADM-v2 compared EDM not significant. Considering the fidelity advantage of ADM-v2 in full-horizon roll-outs, this slight cost in efficiency is acceptable.
4.2 OFF-POLICY EVALUATION
We conduct the Off-policy Evaluation (OPE) on the DOPE (Fu et al. 2021 benchmark and use three metrics, i.e., normalized absolute error, rank correlation, and regret @ 1, to evaluate the consistency between the estimated returns and the true returns of given policies. The detailed definition of these metrics is described in Appendix G.4 We let the policy make full-horizon roll-outs in EDM, RDM, P-Dreamer, ADM, and ADM-v2 to estimate the return and calculate the metrics, respectively. Besides, compare ADM-v2 for OPE with 5 model-free algorithms, including Fitted Q-Evaluation (FQE)
7

---

Published as a conference paper at ICLR 2026
Mean Normalizeed Absolute Error
Mean Rank Correlation
Mean Regretgo1
0.6
0.6
0.5
0.4
9.3
0.2
0.1
0.0
Vis
0.0
2015
Figure 4: The overall OPE results across 15 DOPE tasks.
(Le et al. 2019), Doubly Robust OPE (DR) (Jiang & Li 2016), Importance Sampling (IS) (Kostrikov Nachum 2020). DICE (Yang et al., 2020). , and Variational Power Method (VPM) (Wen 2020). We select is datasets from Hopper, HalfCheetah, and Walker3d in the DOPE benchmark, each with five dataset types. The overall OPE results across these 15 DOPE tasks are shown in Figure averaged over 3 seeds. The exhaustive experimental data be found in Appendix G.4
In terms all three metrics, ADM-v2 for OPE significantly outperforms the five model-free OPE algorithms and also surpasses the performance of other dynamics models for OPE. This result demonstrates the advantages ADM-v2 for OPE and, at the same time, indirectly highlights its strong predictive capability dynamics model.
4.3
OFFLINE RL PERFORMANCE
Table I: Normalized scores after offline learning on D4RL MuJoCo tasks.
Task Name
Model-free
Medel-baad (thort branched roll-out)
Model-based (full-borizon rell-cat)
CQL
EDAC
MOPO
MOBILE
MOREC
ADMPO
MOPO fh
RMPO-th
PDPO-b
ADMPO-Ih
ADM2PO-t (ours)
hopper rand
53
25.3
31.7
31.9
32.1
32.3
31.7
33.3
0.9
20,9
42.7 3.4
Ialfdhettb-nnd
3133
25.4
38.5
30.3
51.6
45.4
26.6
51.1
577
27.0
46.3 1.0
waiker2d-raad
54
16.6
17.9
23.5
22.2
2.2
27.9
0.0
0.0
27.0 2.5
hepee-maa
6199
101.6
62.8
106.6
107.0
107.4
104.2
52.1
13.6
64.8
107
balfchorab-med
469
65.9
73.0
74.6
82.3
72.2
-1.0
62.4
-04
62.7
85. 3.8
us wnikerdd-med
79.5
92.5
84.1
63.7
89.9
93.3
27.0
8.9
-02
58.7
111 0: 4.8
hopper-modne
8633
101.0
103.5
103.9
105
105.4
96.2
76.0
23.9
83.4
107 0 0.5
haltchettb mod
45.3
61.3
72.1
71.7
76.5
67.6
0.82
62.3
7.6
49.0
67.5
walker2d med rep
7688
87.1
85.6
$9.9
95.5
95.6
11.4
6.5
0.3
25.1
106.3
hopper med
96.9
110.7
81.6
112.6
113.3
112.7
11.5
111.3
2.0
110.2
111.7::0.4
bullchertoh-molecyy
95.0
106.3
908
108.2
112.1
103.7
28.6
77.5
1.0
90.7
115.2
maiker2dlmoskcap
109.1
114.7
112.9
115.2
115.8
114.9
100.2
52.7
0.6
83.8
125.1 1
Avcrage
61.6
76.0
70.3
80.0
83.7
81.0
36.5
51.9
4.5
5644
87.0 1.3
Table 2: Normalized scores after offline learning on NeoRL MuJoCo tasks.
Task Name
Model-free
Model-hased (sboet branchied roll-out
Mode-hased (full-horzeon roll-out)
CQL
EDAC:
MUPO
MOBILE
MOREC
ADMPO
MOPO-t
RMPO-B
PDPO-t
ADMPO-D
ADMIPO-DI (murs)
nocdl-luppe--..
16.0
18.3
6.2
17.4
53.5
22.3
24.8
11.2
15.3
17.3
32.3+-3.5
58.2
31.3
40.1
54.7
25.4
52.8
25.4
54.6
20.0
49.1
01.7 2.4
necel noodl-valkkr26-1
44.7
40.2
116
37.6
65.0
55.9
5.2
68.6
30.4
16.6
76.3
3.2
need hopper
46.9
1.0
51.1
31.5
30.4
68.9
8.4
105
halicheetah
77.8
69.3
13.4
walker2d
39.9
62.2
70.1
2.3
H nood--ull mocel-colle 20-H
2.7
Average
We evaluate the policy performance of ADM2PO-fh (full-horizon roll-out in ADM2 for policy optimization) on the D4RL (Fu et al. (020) benchmark and the NeoRL. (Qin et al.) 2022) benchmark. We select 12 D4RL MuJoCo lasks, i.e., random (rand), medium (med), medium-replay (med-rep), and medium-expert (med-exp) datasets from Hopper, HalfCheetah, and Walker2d, and 9 NeoRL MuloCo tasks, i.e., low (L), medium (M), and high (H) levels of datasets from Hopper, HalfCheetah, and Walker2d. We compare the performance of offline RL in ADM-v2 with 2 model-free algorithms, CQL (Kumar et al. 2020) and EDAC (An et al. 202T), and model-based algorithms based on the short branched model roll-out JJnner et al. 2019 MOPO Yu al. 2020), MOBILE (Sun al. 2023).
8

---

Published as a conference paper at ICLR 2026
MOREC (Luo al. 2024), and ADMPO (Lin et al. 2025 Besides, we also make comparisons with the RL performance by making full-horizon roll-out in EDM, RDM, P-Dreamer, and ADM, corresponding to MOPO-fh (EDM), RMPO-fh (RDM), PDPO-fh (P-Dreamer), and ADMPO-fi (ADM). Results averaged over seeds on D4RI. and NeoRI. are shown in Table T and Table
ADM2PO-ft achieves significant advantage in policy performance over previous offline RL base- lines. In particular, compared to the prior state-of-the-arr MOREC, we obtain more than 4.6% performance gain on D4RL and over 12.8% gain on NeoRL. Moreover, among all models, only ADM-v2 can enable the policy to achieve strong performance by making full-horizon roll-outs. Notably, MOPO and ADMPO both suffer from severe performance degradation when conducting full-horizon roll-outs (MOPO-ft and ADMPO-fh), since they lack the capability for long-horizon predictions. The results of ADM2PO-fh demonstrate that when dynamics model possesses the ability for full-horizon roll-outs, it can lead to substantial improvements in policy performance. We further study the influence of roll-out horizon in Appendix G.6
4.4 UNCERTAINTY QUANTIFICATION
423 Corme elen 1,520)
JDM &ie:
IPM Corm pibn: 2-578
10M (Grm olln: 0.5481
Drgampr Narmda on
"anden uctice
tandort otion laamd palky
randor oston
tandar otlan Inamel pakky
randor 20081
luamed ptiky
Itemat paiky
palkyy
rodel
mhode: aetor
mvide anter
mndde mm/
mode mm
Figure 5: Comparison of uncertainty quantification among models on hopper medi V2.
We sample data points in ADM-v2, ADM, EDM, RDM, and P-Dreamer by random action selection, the offline-lemmed policy, and the dataset behavior. For each sample, we compute the model error and model uncertainty, and scatter the results on the hopper-medium-v2 dataset in Figure Results on more datasets are shown in Appendix G. ADM-v2 demonstrates reliable uncertainty quantification, as samples with larger model errors tend to exhibit correspondingly larger model uncertainties. Notably, ADM-v2 achieves a correlation coefficient of 0.928 between model uncertainty and model error, outperforming all other models. This indicates that AAD2 satisfies the assumption of an admissible error estimator Yu et al. 2020). Moreover, ADM-v2 can distinguish different policies better. As samples of the random policy exhibit a clear distributional shift from the hopper-medde medium-v2 dataset, their uncertainties should be higher than the dataset behavior in expectation. Conversely, the uncertainty penalty drives the learned policy toward the safe regions covered by the dataset, where model uncertainty remains lower than the dataset behavior. The scatter plots of ADM-v2 reveal these phenomena more clearly than other models.
4.5 STUDY ON ROLL-OUT HORIZON
A0M 2005
ADM
1BV
RDV
5
10
51
130
5.3
3250
58
303
560
165
26
160
56a
3000
52
153
26
1692
Figure 6: Normalized scores under different roll-out horizons on four D4RL tasks.
To examine the effect of roll-out horizon, we measure the normalized scores obtained by policy optimization in five dynamics models on hopper-medium, walker2d-medium hopper- menpu---ddii tasks under roll-out horizons of (10, 50, 100, 500, 1000) (Figure(6). We find that only ADM-v2 exhibits consistent performance improvements the roll-out horizon increases, while most of the remaining methods either suffer performance degradation or maintain low task. In particular, ADM
and walker2d-medium--eppl
scores as the horizon grows, especially the difficult achieves strong performance under short roll-outs but deteriorates rapidly as the roll-out horizon becomes longer. These results suggest that, compared to other models, ADM-v2 can effectively

---

Published as a conference paper at ICLR 2026
leverage long- nog-horizon roll-outs for better policy learning, owing to its high-fidelity future state predictions.
4.6 STUDY ON HIDDEN REPRESENTATIONS
A2 Hid2n SEEEE (0321100
50
-50
1000
-130
Figure 7: Visualizations of real environmental states and hidden representations in ADM-v2 and ADM, after t-SNE (Maaten & Hinton 2008), on the hopper -e----mdddiiy task.
We select the hopper-medium task and roll out multiple trajectories from the initial state in the real environment, ADM-v2, and ADM, using the action sequence of the dataset. During the roll-out, we record the real environmental states, the hidden states of ADM-v2, and the hidden states of ADM, and then apply t-SNE (Maaten & Hinton 2008) for dimension reduction. We visualize the samples at time steps = 1, 20, 40, 60, and the results are shown in Figure] 7 Since all trajectories start from similar initial states, the samples these four time steps should form four clusters in geometric space. The first plot in Figure Sssows the visualization of real environmental states, where the geometric separation of the four clusters is clear. The second plot shows the hidden states of ADM-v2, which also exhibit well-defined cluster boundaries. In contrast, the fourth plot shows the hidden states of ADM, whose geometric structure appears significantly more entangled. Moreover, by rotating the ADM-v2 hidden-state embeddings by 90 degrees (shown in the third plot of Vigure we observe that the spatial structure of ADM-v2's hidden states aligns with that of the environmental states.
RELATED WORK
This work is related to offline MBRL and OPE. See Appendix H for detailed related work.
MBRL has shown strong potential the offline setting, as it can leverage learned dynamics models to expand datasets and improve data efficiency. To mitigate the issue of accumulated roll-out error, prior works introduce conservative mechanisms and restrict the horizon of roll-outs, including MOPO YYu
al. 2220)). MOReL (Kidambi et al., 2020), MOBILE (Sun al. 2023). MOREC (Luo et al. 2024]. and ADMPO (Lin et al., 2025). 3. In contrast, ADM-v2 introduces a new architecture that enables roll-outs across the full episode horizon, supporting more effective exploration and policy learning. ADM-v2 is an efficient and effective extension of ADM (Lin et al., 2025), which generalizes the direct state transition after executing multi-step action sequence (Asadi et al.) 2018 2019 Che et al. 2018 Machado et al. (023) into an any-step formulation. What's more, multi-step predictions are also considered by Amos al. (2018) Hatner al. (2019).
OPE methods aim to estimate the value of a target policy using only pre-collected data. They are typically categorized into two types. One type model-free methods like fitted Q-evaluation (Le ct al. 2019). importance sampling (IS) (Kostrikov & Nachum, 2020), the doubly robust method (Jiang& & Li 2016), and DICE YYang al. (2020). Another type is model-based methods (Chen et al. 2024b a). Our approach introduces a new dynamics model that proves effective for OPE.
6 CONCLUSION
In this work, we address the long-standing challenge of enabling reliable full-horizon roll-outs in offline MBRL, where existing dynamics models typically suffer from severe error accumulation. We propose ADM-v2 to substantially improve the long-horizon prediction accuracy, enabling both policy evaluation and policy optimization directly with full-horizon trajectories. Experiments on D4RL, NeoRL, and DOPE benchmarks demonstrate that ADM-v2 achieves state-of-the- art performance, surpassing prior model-based and model-free baselines, with more than 4.6% gain D4RL and 12.8% gain on NeoRL. Our results highlight the importance of reliable full- -orihoozzon roll-outs and establish ADM-v2 an effective dynamics model for offline reinforcement learning.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Yangtze River Delta Science and Technology Innovation Com- munity Joint Research Program 2024CSIZN00302, the National Science Foundation of China (2245000,,, and the Natural Science Foundation of Jiangsu (BK20243039). The authors thank anonymous reviewers for their helpful discussions and suggestions for improving the article.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. Proceedings of the 37th International Conference on Machine Learning (0) Virtual Conference, 2020.
Brandon Amos, Laurent Dinh, Serkan Cabi, Thomas Rothörl, Sergio Gomez Colmenarejo, Alistair Muldal, Tom Erez, Yuval Tassa, Nando de Freitas, and Misha Denil. Learning awareness models. In 6th International Conference on Learning Representations (ICLR'18) Vancouver, BC, 2018.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty hhuertainty-bbsed offline reinforcement learning with diversified q-ensemble. In Advances in Neural Information Processing Systems 34 (NeurIPS 21), Virtual Event, 2021.
Kavosh Asadi, Evan Cater, Dipendra Misra, and Michael L. Littman. Towards simple approach to multi-step model-based reinforcement learning. CoRR, abs/18 hass/8811 1.00128, 2018.
Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michael L. Littman. Combating the compounding-eroo problem with multi-step model. CoRR, abs/1905. 11319005..330,, 2019.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, 0ss/1607.064500 2016.
Mohak Bhardwaj. Tengyang Xie, Byron Boots, Nan Jiang, and Ching-An Cheng. Adversarial model for offline reinforcement learning. In Advances in Neural Information Processing Systems 36 (NeurlPS' 23), New Orleans, LA, 2023.
E.F. Camacho and C.B. Alba. Model Predictive Control. Advanced Textbooks in Control and Signal Processing. Springer London, 2013. ISBN 9780857293985. URL http:://bobssss..o googe com. Soom.ecc/boooks11cttttaaaaa
Tong Che, Yuchen Lu, George Tucker, Surya Bhupatiraju. Shane Gu, Sergey Levine, and Yoshua Bengio. Combining model-based and model-free RL via multi-step control variates. https:/ openreview.net 2018. URL
wwt. 1et//orrmm210+HFPP0222
Ruifeng Chen, Xiong-Hui Chen, Yihao Sun, Siyuan Xiao, Minhui Li, and Yang Yu. Policy- conditioned environment models are more generalizable. In Forty-first International Conference on Machine Learning (ICML (4), Vienna, Austria, 2024a.
Ruifeng Chen, Chengxing Jia, Zefang Huang, Than-Shuo Liu, Xu-Hui Liu, and Yang Yu. Offline transition modeling via contrastive energy learning. In Proceedings of the 41st International Conference on Machine Learning (CCML'24), Vienna, Austria, 2024b.
Xiong-Hui Chen, Yang Yu, Zhengmao Zha, Zhihua Yu, Zhenjun Chen, Chenghe Wang, Yinan Wu, Rong-Jun Qin, Hongqiu Wu, Ruijin Ding, and Fangsheng Huang. Adversarial counterfactual envi- ronment model learning. In Advances in Neural Information Processing Systems 36 (NeurIPS'23), New Orleans, LA, 2023.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EM)LP'14), Doha, Qatar, 2014.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31 (NeurIPS' 18), Montréal, Canada, 2018.

---

Published as a conference paper at ICLR 2026
Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179-2) 21179992111,111 1990.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Tom Le Paine. Benchmarks for deep off poolcy evaluation. In 9th International Conference on Learning Representations (ICLR') 21), Virtual Event, 2021.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems 34 (NeurIPS Virtual Event, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning (ICML'19), 19), Long Beach, California, 2019.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The 11th International Conference on Learning Representations (ICLR") 23)) Kigali, Rwanda, 2023.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Omer Gottesman, Yao Liu, Scott Sussex, Emma Brunskill, and Finale Doshi Velez. Combining parametric and nonparametric models for off-policy evaluation. In Proceedings of the 36th International Conference on Machine Learning (ICML' 19), Long Beach, USA, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning (ICML. 18), Stockholm, Sweden, 2018
Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference Machine Learning (ICML' 19), Long Beach, California, 2019.
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learn- ing behaviors by latent imagination. In 8th International Conference on Learning Representations (ICLR" 20), Addis Ababa, Ethiopia, 2020.
Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. 9th International Conference on Learning Representations (ICLR'21), Virtual Event, 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy P. Lillicrap. Mastering diverse domains through world models. CoRR, abs/2301.00 1.01044 2023.
Josiah Hanna, Peter Stone, and Scott Niekum. Bootstrapping with models: Confidence intervals for off-policy evaluation. In Satinder Singh and Shaul Markovitch (eds.), Proceedings of the 31th AAAI Conference on Artificial Intelligence (AAAI'17), pp. 4933-4934, San Francisco, USA, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16), Las Vegas, NV, 2016.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems 32 (NeurIPS' 19), Vancouver, BC, 2019.
Jihwan Jeong, Xiaoyu Wang, Michael Gimelfarb, Hyunwoo Kim, Baher Abdulhai, and Scott Sanner. Conservative bayesian model- based value expansion for offline policy optimization. In The 11th International Conference Learning Representations (CC.R'23), Kigali, Rwanda, 2023.
12

---

Published as a conference paper at ICLR 2026
Nan Jiang and Lihong Li. Doubly robust off off-policy value evaluation for reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning (ICML' 16), New York City, NY, 2016.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In Proceedings of the 38th International Conference on Machine Learning (ICML'21), Virtual Event, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems 33 (NeurIPS' 20), Virtual Event, 2020.
Iyya Kostrikov and Ofir Nachum. Statistical bootstrapping for uncertainty estimation in off-policy evaluation. CoRR, Obs/2007.136099 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems 32 (NewrIPS' 19), Vancouver, BC, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems 33 (NeurIPS' 20), Virtual Event, 2020.
Hoang Minh Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Proceedings of the 36th International Conference on Machine Learning (ICML' 19), volume 97 of Proceedings of Machine Learning Research, pp. 3703-3712 12, Long Beach, California, 2019.
PMLR.
Haoxin Lin, Yihao Sun, Jiaji Zhang, and Yang Yu. Model-based reinforcement learning with multi-step plan value estimation. In Proceedings of the 26th European Conference on Artificial Intelligence (ECAI'23), Kraków, Poland, 2023.
Haoxin Lin, Yan Xu, Yihao Sun, Zhilong Zhang, Yi-Chen Li, Chengxing Jía, Junyin Ye, Jiaji Zhang, and Yang Yu. Any-step dynamics model improves future predictions for online and offline reinforcement learning. In The 13th International Conference on Learning Representations (ICLR"25) 255,, Singapore, 2025.
Xing Liu, Gaozhao Wang, Zihao Liu, Yu Liu, Zhengxiong Liu, and Panfeng Huang. Hierarchical reinforcement learning integrating with human knowledge for practical robot skill learning in complex multi-stage manipulation. IEEE Transactions on Automation Science and Engineering, 2133:3355-38862 2024.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with stationary distribution correction. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference (UAI'20), Tel Aviv, Israel, 2020.
Cong Lu, Philip Ball, Jack Parker-Holder, Michael A. Osborne, and Stephen Roberts. Revisiting design choices in offline model based reinforcement learning. In The 10th International Conference on Learning Representations (ICLR (1CL)'22), Virtual Event, 2022.
Fan-Ming Luo, Tian Xu, Xingchen Cao, and Yang Yu. Reward-consistent dynamics models are strongly generalizable for offline reinforcement learning. In The 12th International Conference Learning Representations (ICLR 24), Vienna, Austria, 2024.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, NNov) 2008
Marlos C. Machado, André Barreto, Doina Precup. and Michael Bowling. Temporal abstraction in reinforcement learning with the successor representation. Journal of Machine Learning Research, 24:80:1-80:69, 2023.
Tianwei Ni, Esther Derman, Vineet Jain, Vincent Taboga, Siamak Ravanbakhsh, and Pierre-Luc Bacon. Long-horizon model-based offline reinforcement learning without conservatism. CoRR, abs/2512.04341,, 2025.
13

---

Published as a conference paper at ICLR 2026
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. In The Thirteenth International Conference on Learning Representations (ICLR 25), Singapore, 2025.
Doina Precup, Richard S. Sutton, and Sanjoy Dasgupta. Off-policy temporal difference learning with function approximation. In Proceedings of the 18th International Conference Machine Learning (ICML'01) San Francisco, USA, 2001.
Rongjun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near woald benchmark for offiie reinforcement learning. In Advances in Neural Information Processing Systems 35 (NeurIPS 222), New Orleans, LA, 2022.
Mare Rigter, Bruno Lacerda, and Nick Hawes. RAMBO-RL: Robust adversarial model-based offline reinforcement learning. In Advances in Neural Information Processing Systems 35 (NeurIPS New Orleans, LA, 2022.
Anya Sims, Cong Lu, Jakob Foerster, and Yee Whye Teh. The edge-of-reach problem in offline model-based reinforcement learning. In Advances in Neural Information Processing Systems 39 (NeurIPS 24), Vancouver, BC, 2024.
Yihao Sun. Offlinerl-kit: An elegant pytorch offline reinforcement learning library. //github. www.cco///c///eee commyyhhasuun./..cccccammm
https
wwroofffieee---- 2023.
Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman incon- sistency for model-based offline reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning (ICML'23) 23), Honolulu. Hawaii, 2023.
Yihao Sun, Zhilong Zhang, Yang Yu, and Lierre-Luc Bacon. Planning with unified multimodal models, 2026. URL htttss/: htittpps://peenrviiee.. nwt/forumm CCMMFRRRRRe 9MM4
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems (I))S'20), Vilamoura, Portugal, 2012.
Weiwei Wang, Yuqiang Li, and Xianyi Wu. Off-policy evaluation for tabular reinforcement learning with synthetic trajectories. Statistics and Computing, 34:41, 2024.
Junfeng Wen, Bo Dai, Lihong Li, and Dale Schuurmans. Batch stationary distribution estimation. In Proceedings of the 37th International Conference on Machine Learning (ICML'20), Virtual Event, 2020.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments for reinforce- ment learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Jiachen Yang, Jingfei Ni, Meng Xi, Jiabao Wen, and Yang Li. Intelligent path planning of underwater robot based reinforcement learning. IEEE Transactions on Automation Science and Engineering, 1983):1983-199,, 2023.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. In Advances in Neural Information Processing Systems 33 (NeurIPS') 20), Virtual Event, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems 33 (NeurIPS' Virtual Event, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline model-based policy optimization. In Advances in Neural Informa- tion Processing Systems 34 (NeuriPS 21)) Virtual Event, 2021.
Jing Zhang, Linjiajie Fang, Kexin Shi, Wenjia Wang, and Bing- Yi Jing. Q-distribution guided q-learning for offline reinforcement learning: uncertainty penalized vavuu via consistency model. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NIPS 24), NIPS '24 Vancouver, Canada, 2024a.
14

---

Published as a conference paper at ICLR 2026
Zhilong Zhang, Ruifeng Chen, Junyin Ye, Yihao Sun, Pengyuan Wang. Jingcheng Pang, Kaiyuan Li, Tianshuo Liu, Haoxin Lin, Yang Yu, and Zhi-Hua Zhou. WHALE: towards generalizable and scalable world models for embodied decision decision-making, CoRR, abs/2411.05619, 2024b.
Zhilong Zhang, Yihao Sun, Junyin Ye, Tian-Shuo Liu, Jiaji Zhang, and Yang Yu. Flow better: Offline preference-based reinforcement learning via preferred trajectory generation. In The Twelfth International Conference on Learning Representations, 2024c. URL /ttpp://openreviee
Det/Eorum?1dd BG6ERSS
Zhilong Zhang, Yunpeng Mei, Xinghao Du, Hongjie Cao, Haonan Wang, Pengyuan Min, Chenyu Wang, Pengfei Chen, Chenbo Xin, Yijie Wang, Wenyu Luo, Yihao Sun, Yidi Wang, Lei Yuan, Gang Wang, and Yang Yu. Hierarchical value-decomposed offline reinforcement learning for whole-body control. In The Fourteenth International Conference on Learning Representations (6), 2026.
Zheng-Mao Zhu, Xiong-Hui Chen, Hong-Long Tian, Kun Zhang, and Yang Yu. Offline reinforcement learning with causal structured world models. CoRR, abs/2206.01474, 2022.
15

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
This paper employs LLMs solely for refining the content of Sections and focusing on improving clarity, coherence, and conciseness of the presentation.
B USEFUL LEMMAS
Lemma B.1 (TV Divergence of Joint Distributions). Given distributions P Pi y) (((r))((yla) and P2(2, v) = (2(x))2(ylT)) We can bound the total variation divergence between
Pi y) and (2(2, as
Drv ( & Drv (F) + max Dry (P) (P(()))))(P((()))))
(7)
Proof.
DTv (FI(
[P -
Σ (P()))))((
Σ PGGG) Px
(8)
- Pa(l)=) -10 P(a)))
+ 1 Σ (x) F2(2)
(1) -
Ep [Dr (P((0)= Drv (P (p))((()))
< max Drv (P' (P(())=) + Drv (P)
Lemma B.2 (TV Divergence Bound of Rolled-out State Distributions). Suppose the expected TV distance between two k-steep transition distributions is bounded as
max Begm P,, DTV (IP ($ (8am) (2 (5m)) S 8*
for each E f1,2,... m), and the initial state distributions are the same, P1(so) = P2())). Then the TV divergence between state distributions the end of the n-th stride is bounded
Drv (Pv (( (2 S m.
(9)
and the TV divergence between two distributions of the k-th state in the (n + 1)-th stride is bounded as
Drv (P: (5mm+ (2 S ndd + 8t
(10)
(This lemma only considers the first timeline starting from Sp, the bound for other timelines of PARoll can be proved similarly.)
16

---

Published as a conference paper at ICLR 2026
Proof. We first extend the absolute difference of P1 (smm) and (2(5n)).
P (8mm) (2(snm)
P1 (1 ( (1 (8())1)))) - (8 (8 1)m) P (m (mm)
[(n 17m
Pi (s (1 (1 - (2 ( 1mm (2 (5(
Σ
(Snml ((n- P2(8nm 8mm ()m-1)m)) (m
(11)
((mm))m
F2 (S (( 1)m) (Pi (s P2((5( (1 13m)))
Σ Pi 1)m) [P (8um ((n 1)m) Pi2(Snm (n 13m)
8(m-1)m
P2 (2 (m- IP (m)-m))m - PP (m)
Next, we have
= Drv (Pi (I1m) 2
en
Σ
PP (9mm) (2()nm)
9pom
1
Esta 1)m ~P, [[P (smm ()m) (s (s 1)m)
Σ Σ Pa IP| (8 P2 P2 (m
2
9(n-1)m
Eg
Σ
(s (mm=))m)
- P2 (2 (m (mm 17m)
(12)
2
Σ
(s - (2
2
A(x
8mh + Dry (P2 13m 1122((
8Th
+ fn-1
8th < n
1=1
where we define
8h B
Bain-
Σ
(i(0(n-i)m4sl000-1 P2 (1 (6 1)m)
which is upper bounded by 8hh Similarly, we can prove
(Tv ( ( a (2 (smm+*)) S th+1 DTy (PP (8)m))))))))))))) (Fam) S nd + 8
(13)
for each & 21,2, ... m).
Lemma B.3 (Bound of Return Gap). Define the return of a given policy π as
= Epp , R(se, at)
(14)
where R is the given reward function and max,,a R(s, a) = -maxx The maximum TV divergence of the multi-step transition distributions denoted as
6max
max
gh
(15)
kE 4E(12, m
17

---

Published as a conference paper at ICLR 2026
We 1h1 denote the return of under transition Tı and f2 to denote the return of n2 under transition T2. P1 denotes the state-action (state) distribution induced by ιı and Tí, and P2 denotes the itateaaction (state) distribution induced by T2 and T2. Suppose the expected TV distance between T and 22 satisfies
max Drv (TI (als)] < cm.
(16)
the return gap bounded as
2 2yTTmaxmma
2r Ex
M1 7pl S (1 7) -mm)
(17)
Proof. First, we consider the expected return of the former nm steps, which is defined as
nm
Σ x R(((
(18)
t=
That is to say 77
lim n7" The gap between 1/!" and m/ satisfies
22+30
1/2 mm
nm
11m
(n- 1)ml
Σ
Σ
(s), P2(st, (((s,,
1/1
1/2
t=(n 1)++1
Se.
(19)
nm
171
(a-1)m
1)m
1/2
(n 1)ml
+2
Σ
mm Drv Drv (t) at))
t= = 11m+1
By Lemma B.I
we have
(1s( at))
SDTv (P:)(P(s + max (ry(m(als))mmmaull
(20)
<Drv(Pi(s))
+
Hence, after applying the result of Lemma
B. we have
1r7 mm
7/2
(n- 1)m
+ 27 (n-
(n 1 + max ((n- ( + 8 Ex)
1/1
1/2
(n- 1)m
(n-1)m
+ 27 (n- 1)m (ndmax + CT)
Timax
Ri
<
0
1/2
+ Σ
y (6- 1)m (i0max + Ex)
Tmax
1=1
1=1
<2c7Tmax + 2 Σg (6- (d8max GT)
Eg
m
7
Fmax
(21)
k=1
m
nm
=2rmax
Ex + 8max
i7
(1- 1)m
Es
k=1
1=1
-2r
Ex + 8max
2 (1- (1-1)m 1)m
7(1
m
+ E
1mm
1 1-ク
2=1
-2rmax
Gx + Smax
1 1-7m
Σ
in
im + Cx
7m 1
7)
18

---

Published as a conference paper at ICLR 2026
Finally, while tends to be oc, we have
I/
1/2
lim R--20
1/2 1//2"
max
G + 8max y: 1 (1 -7)
1 - ym
Σ
inim
+ +
Σ
1=1
1=1
(22)
mmxx
=a
+ 1-7
(1 (1 pm)
27T Omax
maxtx
- (1- 7m)
1-7
Lemma B.4 (Bound of Return Gap without Model Error). Not considering the model error, we have
max Es P, DTv(P1 ( (15mmllPp (mm S. kem = 8
(23)
We use to denote the return of and to denote the return of T2, under the same dynamics. Suppose the expected TV distance between n and T2 satisfies
max Drv
(als)]l (
S cm:
(24)
the return gap is bounded as
2yr max (1 7)2
2r mmmax
Im 72l S
+
(25)
7
Proof. Using the intermediate result of Lemma B.3 we have
nm
7/2
mm
(n-
1m
1)m
22 2y (n-1)m 1)m
T max (nn- 1)8 + 8t + +cx)
171
7/2
171
(n 1)m
1/2
(1-1) 1)m
+ 2y
(n-1n11p 1)m
Mmax ((n (In-))me+ + kez + 6x)
[//
00
26
(1- 1)m
mmax ((i - 1)m + +
1=1
22xxTmax + 22T
(1-1) 1)m
m
maxx ((i 1)m + k k+1 1)
(26)
m
=2f
E + E
((i 1)m + (6-1m + E
A=1
1=1
-2
6x +
1)m + k) (i-1)m + 6x Σ9 7
1=1 k=1
-2max
Cx +
ty
1=1
Finally, while n tends to be oo, we have
m 722
lim
mm
172
22+00
xc
20
2T mn
2r Em
(27)
max
Enr + Enr
t Ex
+
(1 7)2
1 1-
19

---

Published as a conference paper at ICLR 2026
C
THEORETICAL ANALYSES
Theorem C.I (Performance Bound), Suppose the divergence of the k-step transition is bounded as max E P -
Dry (P (twla//eee/l/ss/t/llollllalmaa/m S 8h
max for each & E (1,2,-.. m)) and 6max (1,2, ..)
8h. P indicates the multi-step transition
distribution induced by f and the policy n, and F indicates the multi-step transition distribu- tion induced by the true dynamics T and the policy The policy divergence is bounded as max, Drv ([(((()) $ ex. Then the return gap is bounded as
(n(x) - f[()] & 27Tmaxxmas
(1 11-714 7) - mm) + Ca
4a
(28)
(1 7)2
1 1-7
Proof.
(n(T)) f)(=))
= (n(T) TT(TT)) + (TTT) - 1i(TT)
(29)
(T(T)) ()()) + (m( (((T)
L
La
For L1, we apply Lemma B.4 by using 8max < me (no model error since (TT) and ()(T)) are both under the true dynamics) and obtain
maxxa
2r 2
I <
+
(30)
7)2
?
For Lz, we directly use the result of Lemma B.3 that is
27 2
mamax
L S
(31)
71 ym)
Adding the bounds of L, and L2 together yields the result.
Assumption C.2 (Admissible Error Estimator). Assume that there exists positive b R such that the following inequality equation 32 holds for any maximum backtracking length m and any E S, a, A.
Drv (To o. at), T( at)) < b- 4ADM2 oz);
(32)
where To.m is the overall conditioned distribution coming from
T- ar)
Σ St-k+1 Oc-kitt-11
- a to-----------s san (33)
1=1
and r (st I+t at k+ 1+ttl denotes the distribution over (st-k41:t, a. (41)) conditioned on 8441 induced by the dynamic function T and the policy æ.
Theorem C.3. 8 AMD2 is valid É-uncertainty quantifier, with 6 TTmnx
Specifically,
T-n
-TQ(ss)) an) -DD (DM o1).
(34)
where 1 is the proxy operator induced by ADM-v2 estimate the true Bellman operator T
20

---

Published as a conference paper at ICLR 2026
Proof. First, we define f(+141,fe+1) fet1 + (B (Q( a)] and expand these two Bellman operator to ts Q(s oc)
= me (11sa)
(Becn an ana (u((:+1, (+t1))
A=
P ml -4 4e-m+1t-1/s)
St-m+1 10mm+1dte
k= 8111
8241
Σ D (4 Σ
Tol- (a-arna)l(Gennnee
$1-811 klld
Fr-
f441/111
(35)
and
T=Q(st: az)
(36)
Σ
T(S P144s1
$441.4411
Then, we can obtain
T= - Q(s,
Σ
P.t ac) (es4lss (o(8:41,
fe+1
Σ
1emml 0.) T( P41 ar) (esa(()m)) [((++1,))]
8441
(37)
y max
Σ
(8441, a.) f (241/s;,
mmxx
Drv (Tv ot), T(-le,, at))
Y max ADM2 (DM2 (t))
<b
1
Thus, let 6 TTmax we can say that 3 4ADM2 is valid E-uncertainty quantifier.
21

---

Published as a conference paper at ICLR 2026
D DETAILS OF IMPLEMENTATION
D.1 ADM-V2 LEARNING
We present the pseudo code for learning ADM-v2 in Algorithm !
Algorithm ADM2-learn(Ts, Donv, m. E)
Input: Initial Ta with parameters 0, dataset Denv, maximum stride m, learning rate w, iterations
1: for E iterations do
2: Randomly sample an integer & from [L, ml uniformly
3: Sample (st, - k Se+R) from Done
4: Update ADM-v2 by 0 +- 0 + Vo log To (S Ti+k ara+k-1)
5: end for
6: return T'
D.2 PARALLEL ANY-STEP ROLL-OUT
We present the pseudo code of PARoll (Parallel Any-step Roll-out) in Algorithm] PARoll starts roll-out from given sequence (Sg, dg, 01,"" 8m-2; 2, 0m- 8m-1) sampled from the dataset Donv. At each step, PARoll predicts the diverse states and rewards and estimates the any-step uncertainty in parallel. The parallel computation saves time costs compared to Lin et al. (2055)
Algorithm 2 PARoll To, H, (so, ao 81, ai,"
Om 8m-
Input: Learned ADM-v2 Th with parameters 0, policy Th with parameters o, maximum episode horizon H, initial state-action sequence (so, ao, 81, 01 111 8m Om 8m-
1: bo Roll-out Initialization
2: for i
1 to m do
3: h ency (s;
4: for i to m do
5: (1( = go (h: .4))
6: end for
7:
end for
8:
9:
for t = m - to H do
10:
Choose action by a, T-1(;:) (a, x-1sst ift -1) 1)
11:
Update hidden states in parallel by [h f = lgo (h; am))
12:
Predict diverse states and rewards in parallel by
Kusie 2 Mrst1 2 ))m = [ddec (
(6)
[s(0 [ [N 200 ) (P(() Jmm ~ ( 2
(
20(1)
13:
Uniformly choose the state and reward prediction by
8141 Uniform(s(]. $411 a/m)) and fr+1 Uniform (p(1)
((2)
1+1
Estimate the model uncertainty by u, 4ADM2 (St,
15: Reset the hidden state at the stride edge h,+1
m)+2)
enco ((t mod 2)
841
end for
17:
return (8m-1, am Mm P 8m,
U
D.3 ADM-v2 FOR OFF-POLICY EVALUATION
When the dynamics model has sufficient predictive capability, a straightforward off off-policy evaluation (OPE) method is to treat the dynamics model as data-driven simulator and allow the policy to roll out within it. We let the policy roll out full-horizon trajectories in ADM-v2, and then compute the average return of these trajectories the result of OPF. The pseudo code is shown in Algorithm| 3
22

---

Published as a conference paper at ICLR 2026
Algorithm 3 ADM2OPE(Ts, TG, Deuv. H)
Input: Learned ADM-v2 To with parameters policy n with parameters o, offline dataset Deav. maximum episode horizon H, number of episodes N
1: for to N do
2: Sample sequence (80: ag, r1, 81; 01, r2, ... 8m-22 Om 22, 2, mm-1, ,,, 8m 1) from Denv
3: Model roll-out by PARoll(To, Tg. H, (Su, ag $1, a1 ... 8m-2. am 8 8m-1))
4: Calculate the return by R2 11 7" "A
5: end for
6: return R= N/4 Σド= R
D.4 ADM-v2 FOR OFFLINE POLICY OPTIMIZATION
Roll-out data in ADM-v2 be used optimize the policy offline. We follow the framework of MOPO Yu et al. 2020), but replace the ensemble dynamics model in MOPO with ADM-v2 and set the roll-out length the maximum episode horizon H of the task. The pseudo code of ADM-v2 for
offline policy optimization is shown in Algorithm
Algorithm 4 ADM2PO-fi(TT. To. D Penv E,
Input: Learned ADM-v2 7'a with parameters 0, policy Vy with parameters 6, offline dataset Donv. maximum episode horizon H, model data buffer model. iterations E, penalty coefficient (3
for E iterations do
2: for .M model roll-outs do
3: Sample sequence (so, do T1, $1, 01, T2, ... 5mm am 2, T'm Tm- 8m 8m-1) from Denv
4: Model roll-out by (ARoll(To, TΦ. H, (8n, 4o, 81, (1, 8m Om 8m-1))
5: Penalize the reward by Fg = BL (8t; 04) for each roll-out step
6: Add the penalized model roll-out data Dmodel
7: end for
8: end for
9: for G policy updates do
Update current policy Ty by SAC (Haarnoja et al.) 2018) using samples from D Deny U D model
11: end for return nT
The policy optimization method used in Algorithm 4 SAC (Haarnoja et al. 2018), following MOPO Yu et al. 2020) and ADMPO (Lin et al. 2025).
D.5 NETWORK STRUCTURE
input
Linear
Linexr
Nom
Liner
horm
Linwor
Nom
oulpul
Structure of Incoder/Decoder
Figure 8: The detailed network structure of the encoder and the decoder.
Both the state encoder and the transition decoder of ADM-v2 adopt 5-layer MLP architecture,
where the middle three hidden layers equipped with skip connections (He al. 2016) and layer normalization (Ba et al. 2016). as depicted in Figure 8 The GRU in ADM-v2 does not incorporate any special modifications; it follows the same architecture as in Cho al. (0014) and is implemented directly using torch. nn. GRC
https///ptorchh.org/
23

---

Published as a conference paper at ICLR 2026
E DETAILS OF DYNAMICS MODEL BASELINES
E.1 ANY-STEP DYNAMICS MODEL
dt
ar+1
Qr4k-1
GRU
GRU
GRU
h
Transãion
ADM
Seek
FOOR
Figure 9: Illustration of the first version of Any-step Dynamics Model (ADM), which is structured using a GRU Cell.
Lin et al. (025 propose the original version of Any-step Dynamics Model (ADM). Multi-step dynamics models (Asadi et al. 2018 2019 Che et al. 2018) take s, along with a k-step sequence of actions (ae, 06t1,'-. 0++k-1
imputs to predict $144 and re+k- ADM extends the multi-step dynamics model to allow k to be any positive integer within a specified range, as delineated in
Definition E.
Definition E.1 (Any-step Dynamics Model). Given the maximum stride m, any-step dynamics model T TADM(SS+R: Te+k St, 01:44-11) the distribution of 814- and rt+* E R conditioned on (st, 11: = (st, at 06+1, ((tk-1) E S x AA, where l can be any integer between [1, m].
To handle inputs with variable step sizes, ADM utilizes an RNN (Elman 1990) with GRU (Cho et al. 2014) cell, depicted in Figure 9 Since the input state consists only one step. while the action sequence is of multiple steps, ADM duplicates the state to match the length of the action sequence, and then sequentially feeds it into the RNN. The input (st, M+t++-1).. after being represented by the RNN, yields the hidden h, which is then fed into an MLP to obtain the mean and standard deviation of 8444 and Tt+k- The transition decoder follows the same architecture as ADM-v2, as depicted in Figure 8
Algorithm 5 Roll-out in ADM: ADM-ROlI(TADA. To. H, (sp. $1, $1,01, 8mm-. am 22 8m
Input: Learned 7 ADM. policy T6 with parameters é, maximum episode horizon H, initial state-action sequence (sp, $1, 01," 8mm2, 4m 4mm-2,1 8m-1)
to H do 2: if m then Sample ar n (-18))
1: for
3: else Sample a; T-
4: Randomly sample an integer from [1, m] uniformly
5: if' < m + k - then Roll out via ((441, ~ T Oc
6: else Roll out via ((+1),
(- $1411 a++1-Ma)
7: Estimate the model uncertainty by u, 4ADM (8c. 0e)
8: end for
9: return (sm-1, am 1, Um-11 Pm 8ms 3H 1H
U 1, 81)
With ADM, the frequent bootstrapping during model roll-out can be reduced. Specifically, given the maximum stride m, state-action sequence of length m, (81, 01, 82, 02, 8m am), is sampled from the data buffer to start the roll-out. To obtain the prediction 8m+11 an integer between [1, m] chosen uniformly at random as the backtracking length. k steps are selected for backtracking, (sm-k+1- 4m 4m-k+1mm) will be fed into TADM to obtain the prediction result. After rolling out several steps, it inevitably traces back to previously predicted states. For example, for the prediction 82m. it can only backtrack to one of (sm. 11++1 1: 8m+2, $2m-1). )). The backtracked state, one part of the attribution for the next state prediction, is located several steps ahead, in expectation. Thus, ADM
24

---

Published as a conference paper at ICLR 2026
reduces the actual bootstrapping count of rolled-out trajectory. The pseudo-code of roll-out in ADM shown in Algorithm
The difference among probabilistic predictions ( at (4-1+1;)) obtained with different backtracking l serves natural measure of model uncertainty, which be quantified using variance (or standard deviation), as defined by
UADM (se; ar) E Var (.
(
(38)
E.2
VANILLA RNN DYNAMICS MODEL
Sp 9.)
(St+2) a++1) 1+4R-1100+R-2
GRU
h!
GRU
GRU
lt
Transition Decader
RNN Dynamics Model
S
Prtk
Figure 10: Illustration of the RNN dynamics model, which structured using a GRU Cell.
Vanilla RNN dynamics model shares the same structure ADM but makes the prediction 8 using
the historical state-action sequence (8 0 ... 8t, az 2 as input, as depicted in Figure 10 The transition decoder follows the same architecture ADM-v2, shown in Figure To match the paradigm of ADM and ADM-v2, & is uniformly sampled from [1, ml. Since the predicted current state one part of the input, the prediction error of the next state accumulated based on the deviation of the current state. The bootstrapping is as frequent as the ensemble dynamics model
(Janner et 2019 Lin et al. 2023 Yu al. 2020).
The difference among probabilistic predictions TRNN (-] s Ge-k+1:1 obtained with different also serves as a model uncertainty, which can be quantified using variance (or standard deviation), as defined by
( at) = E
|| Var b - and [eei
(39)
Algorithm Roll-out in RNN: RON-ROLI ue. HI, (so, ap $1,01,-- 8mm-2 0m 9m-1)
Input: Learned TRNN, policy with parameters o, maximum episode horizon H, initial state-action sequence (sp, 4g 81, 01, 88m-2. 2, 0m 8m-1)
for = m
H do
2: if m 1 then Sample a, (-(s))
3: else Sample a ~ (-13:)
4: Randomly sample an integer k from [1, m] uniformly
5: if + t: 2 then Roll out via (2441, (1 - (kx1
6: else Roll out via ($141, f++1) (-NN((1$14 k:t, a+11-A:t)
7: Estimate the model uncertainty by RNN ($r, 0+)
8: end for
9:
return (8m-1, am--1 Mm-1, P 8m:
, 1--1
PH
The pseudo-code of roll-out in RNN is shown in Algorithm
25

---

Published as a conference paper at ICLR 2026
01
ax-1
Ag
GRU
lt
GRU
GRU
h
Zo
Zk
ENC
DEC
DEC
DEC
$0
$1
$1
5k
Sk
Proprioceptive Dreamer
Figure Illustration of the proprioceptive Dreamer, which structured using GRU Cell.
E.3 PROPRIOCEPTIVE DREAMER
Dreamer series (aaner al. 2020 202T 2023) focuses the environment modeling in visual control tasks where the observation is of the image form. We modify the architecture of Dreamer to adapt the proprioceptive control tasks where the observation is just vector, as depicted in Figure 11 Inner modules like GRU, encoder, and decoder share the same network structure as ADM-v2 v2 and other baselines.
The world model in Dreamer consists of a recurrent state-space model that maintains a latent representation
environment. has a prior that predicts the next latent state from the current state and action, and posterior that refines this prediction using the actual observation. A decoder reconstructs observations or rewards from the latent state, while a recurrent core integrates information across time. This latent dynamics model enables the agent imagine trajectories for policy learning.
Algorithm 7 Dreamer-Roll( Dreamer Tg, H, (su, $1, 01, 8m 2, am 8m 9m-1))
Input: Learned Dreamers policy T with parameters d, maximum episode horizon H, initial state- action sequence (3o, do, $1, a1,--- 8m am- 8m-1)
1: for = 0 to 1 do
2: Encode 8 to 2r
3: end for
4: for t = m 1 to H do
5: if
1 then Sample az (-18s)
6: else Sample a ~ (-(8,)
7: Randomly sample integer & from [1, m] uniformly
8: if m + k then Roll via (ě+41, f241) Dreamer (122441 kits a441-kat)
9: else Roll out via (2441,
Dreamer (-12241 knl kka1
10: Decode 2 to $1+1
11: Estimate the model uncertainty by un Dreamer (2,, ac)
12:
end for
13:
return (8m- 4mm-1 Mym Pmy 8mns 8 11 0 -1- Pn, 811)
The pseudo-code of roll-out in proprioceptive Dreamer is shown in Algorithm 7
The difference among probabilistic predictions Trreamer (- -e- lat, az-k+1:t) obtained with different also serves as a model uncertainty, which can quantified using variance (or standard deviation),
It would be (#t-++1; Ot-k+1y"" Sc, 0t) from the dataset if the beginning of the roll-out.
26

---

Published as a conference paper at ICLR 2026
defined by
Dreamer (z,, .4) E V
(e: Beeeemmeeeee te [e+11]
(40)
This uncertainty estimator for Dreamer lies in the latent space.
E.4 ENSEMBLE DYNAMICS MODEL
The ensemble dynamics model (EDM) is applied in many previous model based methods (Janner et al.
2019 Lin et al. 2023 Yu et al. 2020 Kidambi et al. 2020 Sun al. 2023 Luo et al. (024). Given
the ensemble size Nen. Nen MI.P-based single-step dynamics models are learned by maximizing the likelihood, {2) ($++ T++ (e, at) Nan The difference among probabilistic predictions obtained from different learners serves as uncertainty estimation, which can be quantified using variance
J
(or standard deviation), defined by
u EN (st, ai) = Var (o [8+11]]]
(41)
Algorithm 8 Roll-out in EDM: EDM-RolI (T:) (
NeN Th. H, 80)
Input: Learned { policy a with parameters Φ, maximum episode horizon H, initial state so do 2: if 0 then Sample O₁ TT(-ss))
1: for = to
3: else Sample a₂ ~ ns (-18;)
4: Randomly sample integer is from [1, NEN) uniformly
5: if then Roll out via ($:41, Pr41) T (-(s),
6: else Roll out via ($441,f(+1) ~ Th (- a;
7: Estimate the model uncertainty by 2 EN (ōc, ac)
8: end for
9:
return (su, ag: Mo, f1, 81, $1, " 1 UH SH)
The pseudo-code of roll-out in EDM is shown in Algorithm
27

---

Published as a conference paper at ICLR 2026
F EXPERIMENTAL DETAILS
F.1 SETTINGS OFFLINE POLICY OPTIMIZATION AND EVALUATION
We reveal the experimental settings of ADM-v2 for off-poolcy evaluation in Section 4.2 and offline policy optimization in ADM-v2 in Section 4.3 below. Table lists three important aaper--armettrss the maximum stride m, the roll-out horizon H, and the penalty coefficient, under different tasks.
Table Hyper-parameter settings ADM-v2 under different tasks.
Domain Name
Task Name
H
B
hopper-random
1000
100
halfcheotah-random
1000
0.05
walker2d-random
1000
5
hopper-medium medium halfeheetah-min mediiu walker2d meddimm
1000
20
1000
3
D4RL MuJoCo
1000
2.5
hopper-mediumt replay
1000
halfchhctah--medium
1000
5
walker2d-medium- medium-replaae rrplay
1000
0.25
hopper- wwddim-dddppppeeeeee
1000
100
halfcheetah-mediiee medium-experrt
1000
3
1000
2.5
neorl-hopper-L.
1000
25
neorl hallchheethhh--LL
1000
2
woorl-walke22d44
1000
3
NcoRL MuloCo
neorl-hopper-M
1000
20
neor--halfchettah-M
1000
2
neorl- Walker2dMM
1000
3
neorl-hoppers neorl-halfchecta--
1000
25
1000
5
scorl-waller2d-H
1000
3
We set the roll-out length H as 1000, which is the maximum episodic length of MuJoCo Todoroy al. 2012) tasks.
Table
lists other hyper-parameten for learning ADM-v2, kept unchanging under different tasks.
Table 4: Other hyper-parameters for learning ADM-v2.
Hyper-parameter
Value
Description
Lone
number of layers in encoder. size of each hidden layer state encoder.
Hone
200
Laoc
number of leyers in transition decoder.
Bdtec
200
size of each hidden layer in transition decoder.
Lgra
number of GRU layers.
higea
200
size of the hidden state in each GRU layer.
Iradm2
3 x 10
learning rate of ADM-v2,
optimizer
Adam
optimizer of ADM-v2.
Endm2
100
number of epochs for learning ADM-v2. batch size during model learning. batch size during model roll-out. seeds for experiments.
Blearn
1024
B
4096
seeds
(0. 3,
The policy optimization method used in ADM2PO-fh is SAC (Haarnoja et al. 2018). We follow the standard implementation of SAC, with haper-parameters listed in Table 5
28

---

Published as a conference paper at ICLR 2026
Table 5: Hyper-parameters of offline policy optimization in ADM-v2.
Hyper-parameter
Value
Description
No
2
the number of critics.
actor network
FC(256,256)
fully connected (FC) layers with ReL.U activations. fully connected (FC) layers with ReLU activations. target network smoothing coefficient. discount factor.
critic network
FC(256,256)
5 10
y
0.99
Macsor
1 x 10
learning rate of actor.
Teriie
3 10
learning of critic.
optimizer
Adam 256
optimizers of the actor and crities.
batch size
batch size for each update.
Esac
1 x 10
number of steps for policy optimization. sceds for experiments.
sceds
(0, 1, 3, 4)
F.2 SOURCE OF BASELINES' RESULTS
For the evaluation on D4RL (Fu al. 2020 benchmarks, the results of the compared baselines come from three sources:
Retraining on D4RL datasets v2 version with OfflineLL-Kit (Sun 2023), for the algo- rithms whose original papers only report the performance the v0 version, such COL (Kumar et al. 2020), MOPO (Yu et al. 2020).
Including the scores in their papers, for the algorithms whose original papers report the performance the v2 version, such as EDAC (An et al. (202I). MOBILE (Sun et al. 2023)
MOREC (Luo et al. 2024). and ADMPO (Lin et al. (025).
Implementing our code, for policy optimization with full-horizon roll-out in dynamics model basclines, including EDM, RDM, P-Dreamer, and ADM.
For the evaluation on NeoRL (Qin et al. 2022) benchmarks, we report the scores of CQL and MOPO from the original paper of NeoRL, retrain EDAC with OfflineRL-Kit (Sun) 2023), include the scores in papers of MOBILE, MOREC, and ADMPO, and implement the code of policy optimization with hull-horizon roll-out in EDM, RDM, P- P-Dreamer, and ADM.
29

---

Published as a conference paper at ICLR 2026
G ADDITIONAL EMPIRICAL RESULTS
G.1 DIRECT PREDICTION VERSUS BOOTSTRAPPING PREDICTION
hopper-medium-v2
hallcheeeah--ediiu-Z
walker22-medium-v2
ADM-22 m-stap (ours)
3.2
ADM6-2 m-step (ours)
ADI melep (ours)
prediction etror
pred cction error
prediction error
ADM
0.45
boostrapping
2.4
bcotstranping
boststrapping
e*11 -YM-
841 y"W
0.30
1.6
0.00
0.0
0.0
5
hopper-mediuee
taniee--dduu---plla--- replay-v2
Taaffhhetth-mmdiiu--eee
walker 2edeeddd-meiimmm-m repplay---epp-22222
0.60
ADM-22 m-stap (ours)
ADM-x2 m-stup (ours)
ADM-y2 m-step (eurs)
prediction errar
prediction error
ADM
0.45
EDM bootstrapping
EDM m bootstranping
EDM m boststrapping
6411 y"W
prediction
6-(2
0.30
1.6
0.15
0.8
0.00
0.0
happer-medium- -epper-meduum-exper--
seffiheetah-mediu--ee eertt-v22
walker meeiiuu--epeer--22
ADM--2 m-step
ADM--2 (ours)
ADM-v2 tours! stop
prediction error
0.45
prediction emrr
stop
EDM m-boostrapping
EDM m-bocttstarpp anping
EDM m-koststrapping
10l1--"Mit -y!
1.8
-yll1
prediction
1.2
0.15
0.00
m
m
honper--andom-v2
halfcheetah-randee
wa walker2d-randome
m-step
m-step
12
m-tep
prediction error
0.32
mestep
Prediction error
mistep
prediction error
m-step
EDM boststrapping
EDM beststraspiig
EDM boststapping
- 7211 -V
0.75
641 - MM1 1
6-(1 - 711 -
0.16
0.50
0.08
0.25
m
m
Figure 12: Comparison between direct prediction and bootstrapping prediction. We plot the m-step direct prediction error curve ADM-v2 and ADM, the m-step bootstrapping prediction error curve of EDM, and the curve of 1-7" with m varying from to 8. Results are averaged over 5 seeds. 1-
We compare the direct prediction error and the bootstrapping prediction error, as demonstrated in Figure 12 The experiment is conducted on twelve D4RL (Fu et al. 2020) MuJoCo datasets. For each dataset, we divide the data into two parts, one for training and another for evaluation. We sample 100 sequences from the evaluation dataset and calculate the average eтror for each m. We can find
a proper m, with which 6max = max (1.2,-...m) .m] 8² ('11-"") is satisfied, from the evaluation results of almost all datasets. While setting m as the proper value, the performance bound of ADM-v2 tighter than that of single-step dynamics models such as EDM. Furthermore, the direct prediction error of ADM-v2 is smaller than that of ADM and the bootstrapping prediction error of EDM in most cases.
1-7
30

---

Published as a conference paper at ICLR 2026
G.2 ROLL-OUT ERROR
We compare ADM-v2 with four dynamics models in terms of the growth curve of the compounding error (in log scale) as the roll-out horizon increases the maximum episode length, as demonstrated in Figure [3 The experiment is conducted on twelve D4RL. (Fu et al., 2020) MuJoCo datasets. For each dataset, we divide the data into two parts, one for training and another for evaluation. We sample 100 trajectories from the evaluation dataset and execute actions sequentially in each trained dynamics model calculate the average roll-out error at each step. We observe that the roll-out error curve of ADM-v2 is below the curve of other baselines in most cases.
hopper-medium-v2
halfcheettah-medlum-2
10a
10a
10*
roll-out error
roll-out error
roll-out error
108
100
100
ADN-v2 (oura)
ADM-22 (ouns)
ADH-v teurs)
ADN
ADM
ADH
10-2
EDN
10-2
EDM
10-2
TOM
RDN
RDM
RDM
P.Oroamer
10
10
roll-out harizan
rall-oou horizon
roll-out harizon
hopper-medium-reellayv
haaffchettah-meeii---eeaaaaaw
walkker2d-medium-repaaay
10
10a
10
roll-out error
10a
roll-out error
10a
roll-out error
10a
100
10a
109
ADM
10-2
EDN
10-2
EDM
10-
10-4
10
10
500
1000
5000
1000
5000
1000
roll-out horizon
rall-out horizon
roll-out harizon
hopper-mediuum-expertt
hallcheetah-mellum-xxorrer
mallker2-medilmm-expettvvv
roll-out error
10a
roll-out error
10a
roll-out error
10²
10-2
10-2
10-2
RDN
RDM
RDM
P-Draamer
morraimor
10-4
10-1
10
500
1000
0
5.00
1000
500
1000
roll-out horizon
roll-out horizon
roll-out horizon
hopper-random-v2
halfcheetah-randoom-22
walker2ddraadoom-22
100F
10a
roll-out error
10a
roll-out error
10a
roll-out error
10a
100
ADW-2 Jounal
10a
ADHx2 counsi
10"
ADNv2 (euno)
ADM
ADM
ADM
10-2
EDM
10-2
EDM
10
EDM
REN
RDM
KOM
Drccmmor
Drosmmo
Dreamer
10-1
10-4
10-4
500
1000
5.00
1000
0
500
1000
roll-out horizon
roll-out horizon
roll-out horizon
Figure 13: Growth curves of the compounding error (in log scale) as the roll-out horizon increases to the maximum episode length. We compare ADM-v2 with four baselines, including ADM, EDM, RDM, and P-Dreamer, on twelve D4RL MuloCo datasets. Results are averaged over 5 seeds.
G.3 ROLL-OUT EFFICIENCY
We compare ADM-v2 with four dynamics models in terms of roll-out throughput (the average number of roll-out samples per second). Since the state and action dimensions of different types of datasets in each MuloCo Todorov et al.] (012)) task are the same, the corresponding dynamics models also have the same size. Therefore, only conduct experiments the medium-type datasets of three MuloCo
31

---

Published as a conference paper at ICLR 2026
hopper-medium-v2
halfcheetah-medlum-V
walkerrd-mediim--
samples/second
125000
samples/second
125000
samples/second
125000
100000
100000
100000
73000
75000
50000
50000
25000
25000
25000
ADM-2
ADM
IDM
ROM
P-Greamer
ADWA2
ADM
EDM
RDM
Pormeer
ADM-2
ABM
EDM
RDM
P-Dre mar
dynamics models
dynamics models
dynamics models
Figure 14: Comparison of ADM-v2 with ADM, EDM, RDM, and P-Dreamer in terms of roll-out throughput on three D4RL MuJoCo datasets. Results are averaged over 5 seeds.
tasks. The results demonstrated in Figure 14 The roll-out throughput of ADM-v2 is greater than other RNN-based dynamics models, including ADM, RDM, and P-Dreamer. EDM achieves high time efficiency in roll-out due to its simple model architecture. However, the disadvantage of ADM-v2 compared to EDM is not significant. Considering the fidelity advantage of ADM-v2 in full-horizon roll-outs, this slight cost in efficiency is acceptable.
Table 6: Model capacity and GPU memory consumption during roll-out.
Model Capacity
GPU Memory Consumption
Task Name
ADM-v2
ADM
EDM
RDM
P-Dreamer
ADM-v2
ADM
EDM
RDM
P-Dreamer
hopper
1.57MM
0.779M
1.798M
0.779M
1.163M
4.40
4.4G
4.8G
4.4G
4.5G
halfcheetah
1063M
0.787M
1857M
0.787M
L.69M
4.7G
4.6G
5.1G
4.6G
4.8G
walker2d
1.033M
0.787M
1.857M
0.787M
1.16MM
4.8G
4.7G
5.1G
4.7G
4.9G
We list the model capacity and GPU memory consumption during roll-out for each dynamics model in Table/6 Due architectural changes, ADM-v2 has larger number of parameters than ADM, which indeed increases the computational cost of a single forward pass. However, by leveraging the PARoll algorithm designed for ADM-v2, the model removes state backtracking and significantly reduces the number forward passes required during roll-out. As a result, ADM-v2 achieves higher roll-out throughput compared to ADM. terms of GPU memory consumption, ADM-v2 does not require additional resources.
G.4 OFF-POLICY EVALUATION
conduct the Off-policy Evaluation (OPE) experiment on DOPE (Fu et al., 2021) benchmark. The DOPE benchmark provides some policies for each task and proposes several metrics to measure the consistency between the estimated returns by the OPE method and the true returns. We denote N policies given by each task as { T2, ... .TN }. The estimated return by the OPE method for æ, is denoted as T.)) The true return of n, is denoted as nT(T,). We list the definition of these metrics here.
Raw Absolute Error is the average absolute enror between the estimated returns and the true returns of [T1, T2. Tx):
RawAbslErr
1 N
Σ In(m.) -(T)))
(42)
Normalized Absolute Error is the average absolute enror after max-min normalization between the estimated returns and the true returns of [T1, "2,
J:
TT(T)) min, (TT)) max; )(T,) min, 7(TT)) max, T(TTT miny ((,z)
n(T:)) min, )TTT))
RawAbsErs
(43)
32

---

Published as a conference paper at ICLR 2026
Table 7: Raw absolute error of each OPE method on DOPE benchmark, averaged over 3 seeds.
Model-free
Model-based
Task Name
FQE
DR
IS
DICE
VPM
EDM
RDM
P-Dreamer
ADM-v1
ADM-v2 (ours)
hopper-xxp exp
282
426
106
259
442
295
203
210
89
53 ::
hopper-med-cn
252
234
360
bopper-med
283
307
405
215
433
303
322
294
292
293 =
hopper-med-rep
295
298
438
398
444
76
107
319
90
40
bopper-rand
261
289
412
122
438
324
393
276
272 4:
halfcheetal-oxp
1031
1025
1404
945
1087
1049
821
1011
915 4: 61
halfchectah-medd med-exp
1014
1015
1400
1078
1427
1184
961
872
852
711 + 108
halfcheetah-mon
1211
1222
1217
1382
1374
969
891
1343
874
778 = 54
halfchectah- med-rp
1003
1001
1409
1440
1384
1009
1063
1442
1020
873 73
balfcheetab-rand
938
949
1405
1446
1411
1001
809
1527
955
856 + 25
walker2d-exp
453
519
405
437
367
458
337
398
250
228 =: 32
walker2d med-exp exp
233
217
436
20
walker2d-med
350
368
428
273
426
393
386
367
213
204 =: 46
walker2d-med-rep
313
296
427
374
424
358
379
372
194
192 = 22
walker2d-rand
354
347
430
466
407
350
368 :: 20
Average
552
568
712
625
720
549
626
452
1: 33
Table 8: Normalized absolute error of each OPE method on DOPE benchmark, averaged over 3 seeds.
Model-free
Model-based
Task Name
FQE
DR
IS
DICE
VPM
EDM
RDM
P-Dreamer
ADM-r1
ADM-22 (ours)
hopper-exp
0.55
0.83
0,21
0.51
0.86
0.58
0.40
0.41
0.18
0.10
hopper-med-exp
0.46
0.70
0.52
0.61
0.20
0,46
0.15
0.07 0.011
hopper-med
0.55
0.60
0.79
0.42
0.85
0.59
0.63
0.57
0.57
0.58 + 0.01
hopper-med-rep
0.58
0.58
0,86
0.78
0.87
0.15
0.21
0.62
0.17
0.08 : 0.02
hopper-rand
0.56
0.24
D.77
0.54
0.02
halfcheetah-exp
0.49
0.49
0.45
0.52
0.50
0.39
0.48
0.44
halfcheetah med-vee exp
0.49
0.49
0.67
0.52
0.68
0.57
0.46
0.42
0.41
0.34 +: 0.05
halfcheetah-nee
0.58
0.59
0,58
0.66
0.66
0.46
0.43
0.64
0.42
0.37 0.03
halfcheetah rep
0.48
0.51
0.49
0.42 0.04
halfcheetab-rand
0.45
0.45
0.67
0.69
0.68
0.48
0.39
0.73
0.46
0.01
walker 2d-ee2ddeee exp
0.86
0.99
0.77
0.83
0.70
0.87
0.64
0.76
0.48
0.43 0.06
0.44
0.41
0.83
0.61
0.81
0.85
0.65
0.73
0.46
0.36
walker2d-me 2ddd
0.70
0.52
0.75
0.73
0.70
0.40
0.39
2d-kd-rrdd
0.59
0.56
0.81
0.71
0.80
0.68
0,72
0.71
0,37
0.37
walker2d-rand
0.67
0.66
0.82
0.80
0.84
0.88
0.69
0.77
0.67
0.00
Average
0.56
0.58
0.70
0.60
0.75
0.61
0.52
0.62
0.42
0.37
Table 9: Rank correlation of each OPE method on DOPE benchmark, averaged over seeds.
Model-free
Model-hased
Task Name
FQE
DR
18
DICE
VPM
EDM
RDM
P-Dreamer
ADMsr1
ADM-22 (ours)
hopper-exp
-0.33
-0.41
0.37
-0.08
0.21
0.46
0.67
-0.26
0.83
0.90
bopper-med-exp
0.01
-0.08
0.35
0.08
0.17
0.54
0.83
-0.25
0.90
0.95 0.01
hopper-med
-0.29
-0.31
-0.55
0.58
0.51
hopper-med-rep
0.45
0.05
-0.16
0.27
-0.16
0,91
0.93
-0.23
0.93
0.97 = 0.01
hopper-rand
-0.11
-0.19
0.23
-0.13
-0.46
0.35
0.39
-0.20
0.40
0.51 0.10
halfcheetah exp
0.78
0.77
0.01
-0.44
0.18
0.51
0.53
0.71
0.56
0.87 0.10
halfchectah-medde
0.62
0.62
-0.06
-0.08
-0.47
0.21
0.87
0.65
0.88
0.94 0.02
halfcheetah med
0.34
0.32
0.80
0.26
-0.35
0.37
0.86
0.87
0.92
0.94 1: 0.01
aaffcheetah-med-rre
0.26
0.32
0.59
0.71
0.71
halfcheetali rand
-0.11
-0.02
-0.24
-0.70
0.27
0.75
0.81
-0.58
0.63
0.78 : 0.06
walker2d-exp
0.35
0.26
0.22
-0.37
0.17
-0:05
-0.09
-0.51
0.09
0.15 0.11
walker2d med-exp exp
0.25
0.19
0.24
-0.34
0.49
0.21
0.27
00.8
0.61
0.70 0.10 0.10
walker2d-med
-0.09
0.02
-0.25
0.12
0.37
0.03
-0.69
0.56
0.68
walker2d med-rep
-0.19
-0,37
0.65
0.55
-0.52
0.14
0.23
-0.45
0.65
0.75 1: 0.03
walker2d-rand
0.21
0.16
-0.05
-0.19
-0.42
-0.04
0.44
-0.21
0.44
0.54 0.11
Average
0.14
0.09
0.14
-0.10
-0.03
0.40
0.56
40.05
0.67
0.77 0.05
Rank correlation measures the corelatiin between the ordinal rankings of the estimated returns and the true returns:
RankCorr Cov([ι(T;))I (
(44)
33

---

Published as a conference paper at ICLR 2026
Table 10: Regret@1 of each OPE method on DOPE benchmark, averaged over 3 seeds.
Model-free
Model-bused
Task Name
FQE
DR
18
DICE
VPM
EDM
RIM
P-Dreamer
ADM-y1
ADM-v2 (ours)
exp
0.41
0.34
0.06
0.20
0.13
0.07
0.10
0.49
0.09
0.05 +: 0.02
hopper-med-epp
0.42
0.34
0.10
0.16
0.12
0.14
0.07
0.58
0.06
0.06
hopper-med
0.32
0.32
0.38
0.18
0.10
0.42
0.16
0.27
0.06
0.08 0.02
hopper-med-ee meeper-ed--eep rep
0.18
0.34
0.88
0.16
0.21
0.16
0.04
0.28
0.06
0.03 +: 0.04
hopper-rand
0.36
0.41
0.05
0.30
0.26
0.40
0.52
0.57
0,51
0.49 - 0.20
balfcheetah-exp
0.12
0.11
0.15
0.32
0.30
0.25
0.35
0.23
0,18 0.15
maffchetthh-med-eee
0.14
0.14
0.73
0.38
0.80
0.36
0.10
0.48
0.11
0.02 0.04
halfcheetah med
0.38
0.37
0.05
0.82
0.33
0.17
0.17
0.19
0.16
0.13 + 0.09
0.36
0.33
0.13
0.30
0.23
0.18
0.16
0.14
0.06 0.133
balfcheetah-ran rand
0.37
0.31
0.31
0.81
0.12
0.41
0.20
1,00
0.32
0.24 0.02
walker2d-exp
0.06
0.06
0.43
0.35
0.09
0142
0.49
0,90
0.56
0.44 + 0.32
walker 2d med-exp
0.22
0.30
0.13
0.78
0.24
0.43
0.10
0.80
0.04
0.02 + 0.02
walker2d-mer
0.31
0.25
0.70
0.27
0.35
0.36
0.80
0.13
0.04
walker 2d memeedrrep
0.24
0.68
0.02
0.18
0.46
0.21
0.12
0.77
0.08
0.04 - 0.00
walker2d-rand
0.15
0.15
0.74
0.39
0.88
0.57
0.12
0.50
0.03
0.03 0.02
Average
0.27
0.30
0,32
0.37
0.28
0.31
0,20
0.54
0.17
0.13 : 0.07
where Cov(:,.) is the covariance between two sequences, and o(.) is the standard deviation of sequence.
Regret@l @K k is the difference between the return of the best policy in the entire set and the return of the best policy in the top-k set (where the top-k set is chosen by estimated returns):
Regret@ak
max (TT))))
max T7(T));
(45)
sE1:N
jEtopk(I:N)
where topk(1 N) denotes the indices of the top K policies as measured by estimated returns (TT)). We use regret@) in the experiment.
We report the raw absolute error. normalized absolute error, rank correlation, and regret@ of each OPE method on the DOPE benchmark in Table/7/ableBBaaabll 9 and Table I0] respectively. We select 15 datasets from Hopper, HalfCheetah, and Walker3d, each with five dataset types: expert (exp), medium-expert (exp), medium (med), medium-replay (med-rep). and random (rand). The results of FQE (Le al.) 2019). DR (Jiang & Lij (016). (Kostrikov & Nachum, 2020), DICE (Yang et al., 2020), and VPM (Wen et al. 2020) come from the DOPE (Fu et al. 2021) benchmark.
G.5 OFFLINE POLICY LEARNING WITH OFF-POLICY EVALUATION
We plot the true return curves obtained from the real environment and the estimated return curves provided by dynamics models during offline policy learning, as shown by Figure [5] Our results reveal that only ADM-v2 produces estimated return curves that closely align with the true returns. Since the real environment is not accessible during offline policy optimization, the accuracy of model-estimated returns becomes crucial. Reliable return estimates are essential both for monitoring policy convergence and for selecting the final policy to be deployed in the real world. As shown in the figure, ADM-v2 satisfies this requirement, highlighting its significance for advancing the practical application of reinforcement learning in real-world scenarios.
G.6 STUDY ON ROLL-OUT HORIZON AND PENALTY COFFFICIENT
We evaluate the policy performance after learning ADM-v2, ADM, EDM, RDM, and P-Dreamer on the hopper-medium-v2 task under different roll-out horizons and penalty coefficients and plot the heat map in Figure 16 While setting the roll-out horizon to 1, all models produce policies with poor performance. As the roll-out horizon is gradually increased up to the maximum episode length, the policy performance learned within ADM-v2 tends to improve after appropriately tuning the penalty coefficient. This highlights the potential of full-horizon roll-outs in enhancing policy learning. However, this trend is not as evident for other models, since their predictive accuracy deteriorates in full-horizon roll-outs and thus fails provide reliable support for policy optimization.
34

---

Published as a conference paper at ICLR 2026
hopper-medium-replla
hopper-mellmmreplly
4000
ADN V2 (purs)
EDM
Drrcmmar
model estimated return
4000
ADM o2 (ours)
EDN
P-Dreomer
ADN
RDM
ADM
RDM
rea retur
3000
3000
2000
2000
1000
10000
2000
epochs
epochs
walkeer2d-medium-reeee
walker2d wwllkerzommddlumreppla
6000
6000
ADN v2 (ours)
EDM
Dreamar
model estimated return
ADN #2 (ours)
EDM
Drcamer
ADN
RDM
ADM
RDM
2ea retur
4000
4000
2000
2000
1000
1000
1000
1000
4000
epochs
epochs
Figure 15: Real return curves vs estimated return curves by dynamics models during offline policy learning on peappermediumm-rppe and walker2d-medinee mediim--reddlayy tasks, averaged over seeds.
ADM-v2 (ours)
ADM
EDM
RDM
P-Dreamer
peraltt coofiiciont
6.3
10
28
13:04
10-24
100758 horizon
13.22 paizerize
28 200
cill-outhe harizon
out horzon
horizon
26
49
60
59
100
Figure 16: Heat map of policy performance after learning in ADM-v2, ADM, EDM, RDM, and P-Dreamer on hopper-men reddi um-v2, under different roll-out horizons and penalty coefficients.
G.7 ABLATION STUDY ON ENCODER STRUCTURE
Table 11: Normalized scores of ablative settings.
Task Name
original
layer less
layer
w/o skip connections
w/o layer normalization
hopper-medium
0.3
106. 0.1
107.2
105.1 0.9
108.0 1: 0.2
melker
111.6 4.8
108.8 1.3
111.8 4 1.1
109.7 +: 1.4
109.2 4 2.5
hopper-medium-replee replay
107.0 0.5
1.7.2
105.2 2 0.8
107.2 +: 0.3
106.6 : 0.2 106.7 1 3.1
106.3 4.0
1.5
107.3
109.5 2.6
The state encoder in ADM-v2 adopts a 5-layer MLP architecture, where the middle three hidden layers are equipped with skip connections (He al. 2016) and layer normalization (Ba et al., 2016). We aim to investigate the impact of network depth, skip connections, and layer normalization on the algorithm's performance. We design four ablative settings:
layer less: Reduce one hidden layer in state encoder.
layer more: Add one hidden layer in state encoder.
w/o skip connections: Remove all skip connections in state encoder.
w/o layer normalization:
Remove all layer normalization modules in state encoder.
35

---

Published as conference paper at ICLR 2026
Table 11 presents the results four tasks: hopper-medium, walker2d-medium hopper- hopper-medium- replay, and walker2d-medium-rree medium-replay. can be observed that the architecture of the state encoder has only minor effect the algorithm's performance.
G.8 ROBUSTNESS TO DATA COVERAGE
On the Hopper-medium-ye walker2d-medium-ye hopper-medium-rellay- replay-v2 datasets, we run the algorithms using 50%, 20%, 10%, and 5% of the data, respectively, and compare their performance with that obtained using the full dataset. The results of MOPO, MOPO-fh, ADMPO, ADMPO-fh, and ADM2PO-IP are reported in Table 12
and walker2d-medium- medium-
Table 12: Normalized scores under different data coverage.
Task Name
МОРО
MOPO-h
ADMPO
ADMPO-th
ADM2PO-fh (ours)
hopper-medium (100% data)
62.8
104.2
107.4
64.8
107.6 0.3
hopper-medium (50% data)
42.4
106.1
100.7
68.5
107.3 + 0.5
hopper-medium (20% data)
43.3
102.3
73.6
48.3
1.4.4-0..8
hopper-medium (10% data)
35.8
56.0
75.1
54.7
103.6 6: 0.7
hopper-medium (5% data)
44.6
39.7
51.3
50.9
102.6 1.116.1..1
walker2d-mediu 2d-mediummmmm (100% data)
84.1
27.0
93.2
58.7
111.6 4.8
walker2d- -medium (50% data)
73.5
34.1
79.5
24.2
109.5 2.5
walker2d-medee (20% data)
55.7
17.0
75,5
30.6
102.4 - 1.4
(10% data)
11.9
10.9
61.3
22.7
99.3 1.7
walker2d-medium (5% data)
10.6
6.6
10.6
-0.5
83.9 1.3
mooper-medium-replay (100% data)
103.5
104.4
83.4
107.0 0.5
hopper-mane saaper-medium-rellay (50% data)
21.7
25.7
25.9
17.1
107.4 1.2
mepper--edium--eplay (20% data)
16.0
20.6
18.1
16.4
98.7 2.5
hopper-medium-tee (10% data)
18.2
10.9
11.9
12.5
96.6 1.8
modper-medium-repllay (5% data)
11.3
12.3
11.7
11.5
12.7 2.4
walker2d-medium--rople medium-replay (100% data)
85.6
95.6
26.1
106.3 4.0
mawiirr2--mmiim-rpppaaa (50% data)
76.3
67.4
23.5
98.3 2.7
walker2d-mediutee medium-replay (20% data)
58.7
21.6
94.9 3.1
walker24-medium-ee medium---pmi (10% data)
8.8
-4.5
15.9
7.8
86.8 2.5
walker2d medium-rraa data)
3.5
-3.1
16.9
7.9
As the amount of available data decreases, the performance of algorithms tends to decline. Overall, ADM2PO-th shows the smallest degradation, demonstrating its robustness to data coverage. Notably, on the Sopper-medium-i and walker2d-mcdium- v2 datasets, ADM2PO-fh still learns policies with great performance when trained with only 5% of the data.
G.9 ROBUSTNESS TO ENVIRONMENTAL PERTURBATIONS
Table 13: Normalized scores under environmental perturbations.
Tack Name
original
state noise (000)000))
state moise (V0,0.00
state noise A'0,0.11
smaller gravity (0.9g)
greater gravity (1.lg)
hopper medium
107.6 0.3
107.0 0.5
106.2
20,0 3.11
61.2 1.22...5 5.5
38.5 L 3.7
walkerdd medium
1.1. 1: 4.8
111.1
111.11 4.
76.77 7.6
73.2 1 6.2
08.5 = 8.4
syper-meiimmepppe
10.00 0.5
106. 106.51:00.
105.7 2 06
35.0 7.9
78.5 4.7
$1.1 5.5
wilker2d medium replay
106.3
105.9 2.4
105.5 2 5.3
67.2 6.9
91.61 8.7
76.0 : 9.4
Table 13 reports the performance of the policies learned by ADM2PO-fh on the four datasets, hopper- seneeer-mdduu--ppla--eee and walkerzö-meddiu---eplaa
medium-v2, valker2d-medium--22 under different environment perturbations. We consider five types of perturbations: adding Gaussian noise with a standard deviation 0.001, 0.01, and 0.1 to the environmental states, decreasing the gravity to 0.9g, and increasing the gravity to 1.1g. As shown in the table, the policy exhibits noticeable performance degradation when the noise level reaches O.I, while maintaining reasonable robustness under the other perturbations.
36

---

Published as a conference paper at ICLR 2026
Table 14: Normalized scores after offline learning on OGBench tasks.
Tisk Nume
MOPO MOBILE
ADMPO
MOPO-h
RMPO-t
PDPO-h
ADMPO-th
ADM2PO-t (ours)
cube singl-ingleegne play singlesng
72
65
0
11
77
cube-douhle-phin whauill-ply-sinnletta singletask
12
0
35
scene play singletakk
6
11
0
0
0
3
22 G
muzzle-3s3-plee play sing-sing
18
puzzle-Ixh-e ee--lay--inggliat
0
0
0
0
13 8
G.10 OGBENCH RESULTS
Table 14 reports the normalized scores of ADM2PO fhh and several other model-based methods on
OGBench (Park et al. 2025) tasks. Most tasks in OGBench are designed for goal-conditioned offline RL algorithms,, making them less suitable for the method proposed in this paper. Therefore, we select five single-task manipulation datasets from OGBench to evaluate ADM2PO- As shown, on these challenging hong-horizon manipulation tasks, ADM2PO-fh achieves better performance than other offline MBRL algorithms.
G.11 UNCERTAINTY QUANTIFICATION
ADDM222 (Commtation: 9220
AD (Comeation: ó 871)
EOM (Come 0.576)
18-
RGM (Cormlation: 2.500)
D-Dmeamter (Comaletion 3-183
actior datamu:
ransomacion datagun
ransomaciinn
dotmett
poveeeon May
dataaet
ditasst
myde upcee
mide
mode erer
model antr
model
model orrer
model wrer
Figure 17: Comparison of uncertainty quantification on hopper-med u2
rendem aethor learred soliy
rintom artior Isamnal by
ransomaction
rantemaatton
nendin actinn
Isemaal polky
Isemaa porky
burnad by
mode: enree
mode cmee
medel entr
medel emer
moder entor
Figure 18: Comparison of uncertainty quantification on hopper medi um rep replay v2.
ADN (Commtaton:
(Cameaton:: (917)
(ormealionn 515)
BBM (Crrmbetin 5.520)
P.Dowamee (Comaleion:
candom oction learned sol dotmett
Cansem action
ransemaction
Yandom action lsamed polcy datase:
random actinn komed go
Ieamed pel detsmet:
leamed polay
datamo:
datasst
inced
mode soowtte
21
12
12
Figure 19: Comparison of uncertainty quantification on hopphr um wwweepeeeteeee..emmm V2.
We sample data points in ADM-v2, ADM, EDM, RDM, and P-Dreamer by random action selection, the learned policy, and the dataset behavior. For each sample, we compute the model error and model uncertainty, and scatter the results in Figures 17 to 25 ADM-v2 demonstrates reliable uncertainty quantification, as samples with larger model errors tend to exhibit correspondingly larger model uncertainties. Notably, ADM-v2 achieves great correlation coefficient between model uncertainty and model error, outperforming all other models. This indicates that 4ADD2 satisfies the assumption of an admissible error estimator (Yu et al. 2020
37

---

Published as a conference paper at ICLR 2026
ADM Correbrion D 1821
EDM Corraclon 2.507)
RDM Commaton 2514
comner Komeation
octaon
random odter
ransom adion
tandom actin
randem
2.5
Inarned rel detssee
leamed relry dutsor
iramed pelry
komed pairy
bomes kol
urcement
moooe urreet unremnaityy
abtert
R
Figure 20: Comparison of uncertainty quantification on walker2d-medi V2.
ADM (Comabeien:
AGH covrebee
FOM Comellea:: 5 (.00)
BDN CComeaation: 2,505)
Dreamer [Comeation on 0.420
candem octir
ransom adton
raneom adtinn
02
ang
midd ofee walker2c medi um replayy-a....... v2
moed ondt
Figure 21: Comparison of uncertainty quantification
ADM Comeelltenn
EDM (Camalelee::
RDN (Comelater: 2.525)
Dreammr (Comeation:
oction learnaal col
randemuctio leatnall pet
ranzom adkan icemaal polay
ureeom ection lamall juay
tanebrr ecion hamaal go tbber
2.5
25-
datsor
dsteer
abter
mooee inrern
modeleem:
mostteroo
mode
13
Figure 22: Comparison of uncertainty quantification on wa walker2d medi um -etttme--eppeeee v2.
ADW-22 (Cometeler: 5,000)
ADM CCamatatie:: 2.634)
COM (Comsteton: 2.555)
SDM (Commtation: D 500)
P-Oreamer (Core alten: o 3041
randonn
mode unodamy
dakasci
dalanst
colaast
cobust
model wrarr Figure 23: Comparison of uncertainty quantification halfcheetah www.heeeeta-meeiiiie um 2.
mocel
model erter
midel wrrer
midel orrer
(Conslatoos 8819)
(Camelatons 3.763)
EDM (Conelatoos 0.539)
RDN (Coredalion: 0.4g))
P.Orsumet (Corre ution: O
rincem action learred polcc datasit
ran dom action carnel po ky datasut
ran den acter sarnal po ki cabasst
randen act on koama carnol polkyy cataest
rendim action loames palky catasst
modnl incervitty
3 12
12
model
Figure 24: Comparison of uncertainty quantification on ha 1 www.ffeeeehhhmee um rep v2
ADM 42 cComearion, 6.855)
ADM Conreratonn 8.78
EDM rooreno 6.364)
RDM (Coredatin: 6.673)
P.Oroomer (Corre adion 0 0.521
inamed
po datanci
cared
wared
lamed
saluvt
cokest
telustt
modta
mocel GIFO
mozel erer
model eroz
mide eneer
mode enee
Figure 25: Comparison of uncertainty quantification on hal walffeeeeahmmedd www.meewwweeeeeeeeeeee
38

---

Published as a conference paper at ICLR 2026
ADW 22 Commmation 9777
5DM Kanndation 0.57)
EiM Kameation 0.928)
RDM fommbron 2.516)
oction
odter
ransom acton
randem action
random
Inareed nel dechnee
leamed relry Autsor
iramed pelry
iramed polry dateer
Iomed polly
duteer
moooe ureemaiity
X
2- 24
32
Figure 26: Comparison of uncertainty quantification under short roll-outs (10-step roll-out) on Sopper-medium-1
G.12 UNCERTAINTY QUANTIFICATION UNDER SHORT ROLL-OUTS
The uncertainty quantification experiments in Appendix G.11
all conducted under full-horizon roll-outs. Figure 26 presents the uncertainty quantification results on the when rolling out only 10 steps. When the roll-out horizon is reduced, the correlation between the uncertainty estimated by these dynamics models and the model error becomes higher than that under long-horizon roll-outs.
task
H
RELATED WORK
H.1 OFFLINE MODEL-FREE REINFORCEMENT LEARNING
The interactive exploration inherent in online reinforcement learning is often unsafe and expensive for
domains, such as robot control (Liu et al. 2024 Yang et al., 2023 Zhang et al.)
2026). Offline Model-free Reinforcement Learning (OMFRI) (Fujimoto et al.) 2019 Fujimoto & Gu 2021 Kumar al. 2020 Zhang et al. 2024c) seeks to learn policies directly from static datasets
without access to the environment. To prevent distributional shift from divergence with the behavior
policy, OMFRL typically utilizes some form of importance sampling (Precup al. 2001 Liu et al.)
2020). constrains the learned policy through regularization (Kumar et al., 2019 Fujimoto & Gu
2021). ). or discourages unreliable actions by adopting pessimistic Q-value optimization methods, such as uncertainty-awar gAgarwal et al.) 2020 Zhang et al.) (024a) and conservative estimation (Kumar ett al. 2020 Garg et al. 2023). However, these strategies often limit generalization beyond the dataset. approach can extrapolate to unseen state-action pairs, enabling stronger
By contrast, our generalization.
H.2 OFFLINE MODEL-BASED REINFORCEMENT LEARNING
learned environment model can be used for planning (Camacho & Albal 2013) Chua et al. 2018 Sun al. 2026), as well as acting simulator for policy learning (Janner et al. 2019] Yu et al. 2020 Lin et al. 2025). Offline Model-based Reinforcement Learning (OMBRL) has shown strong potential in the offline setting, as can leverage learned dynamics models to expand datasets and improve data efficiency. However, model-generated roll-outs often suffer from distributional shift, which undermines long-horizon predictions. To mitigate this issue, prior works introduce conservative mechanisms and restrict the horizon of roll-outs. MOPO (Yu et al. 2020) and MOReL (Kidambi et al. 2020) penalize rewards with model uncertainty for pessimistic value estimation, while MOBILE inconsistency. COMBO (Yu et al. 2021) applies CQL (Kumar et al., 2020) on model--eeeraane samples to constrain out-of- distribution Q-values. RAMBO (Rigter al. 2022) employs adversarial model learning to enforce conservative estimation, and CBOP (Jeong et al. 2023) integrates adaptive short-horizon rollouts with ensemble variance for robust value estimation. MOREC (Luo et al.) 2024) further enhances reliability by designing a reward-consistens dynamics model.
(Sun et al. 2023) improves uncertainty quantification via
Despite these advances, all methods remain constrained to short medium horizons due to com- pounding model errors. In contrast, ADM-v2 introduces a novel architecture that enables roll-outs across the full episode horizon, thereby supporting more effective exploration and policy learning.
39

---

Published as a conference paper at ICLR 2026
H.3 OFFICOLICY EVALUATION
Off-policy evaluation (OPE) aims to estimate the value of a target policy using only pre-collected data. There several rypical types of OPE methods. Fitted Q-evaluation (FQE) (Le et 2019) estimates a value for the Q-function and then utilizes it as the policy value estimator. Importance sampling (IS) (Kostrikov & Nachum 2020) reweighs observed trajectories by the likelihood ratio between the target and behavior policies. The doubly robust method (Jiang & Li 2016) combines the importance sampling technique with value estimator for variance reduction. DICE (Yang et al. 2020 uses saddle point objective to estimate marginalized importance weights. Variational Power Method (VPM) (Wen et al. 2020) runs variational power iteration algorithm to estimate the importance weights without the knowledge of the behavior policy. Model-based OPE methods (Hanna et al.) 2017 Gottesman et al. 2019, Wang et al., 2024; Chen et al. 2024b a) learn an approximate dynamics model and evaluate the policy by simulating trajectories within the learned dynamics model.
Model-based methods are appealing for their ability to generalize beyond observed data, but their performance heavily depends the accuracy of the learned model, which substantially affect OPE outcomes. Our approach builds on this paradigm and introduces a new model that demonstrates improved effectiveness for OPE.
40


Reviews:
[{"review_id": "UOuaPmru3m", "reviewer": "Reviewer_6xfG", "summary": "The authors propose ADM-v2, a novel dynamics model architecture that builds upon the \"any-step\" direct prediction paradigm of the original ADM. The key innovation is decoupling the initial state from the recurrent processing of the action sequence. ADM-v2 allows full-horizon roll-outs and achieves new state-of-the-art performance on both D4RL and NeoRL benchmarks.", "questions": "[Q1] How does ADM2PO (short, limited horizon rollout of ADM2) perform compared to the reported ADM2PO-fh (full horizon rollout)?\n\n[Q2] How does ADM2PO perform in tasks that truly require long-horizon rollout and planning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "[S1] ADM-v2 achieves high empirical performance while maintaining consistent uncertainty estimation and fast dynamics rollout. It also supports full-horizon expansion, which opens up new possibilities for learning long-horizon tasks.", "weaknesses": "[W1] The evaluation only consists of MuJoCo locomotion tasks, which are mostly reactive. Model-free and model-based methods with short-horizon prediction also achieves high performance on MuJoCo. It is unknown that if full-horizon rollout is required.\n\n[W2] This work is an incremental (albeit highly effective and important) improvement focused on a refined architecture and a new roll-out algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hgVk6A86vz", "reviewer": "Reviewer_BzbB", "summary": "This work creates an extension of the Any-step Dynamics Model (ADM, ICLR2025). This is a model-based RL method that creates a dynamics model that can directly predict m steps forward by feeding in a sequence of the next m actions (there existed some previous works that did this, but the ADM paper made it more practical). The ADM model has a GRU recurrent network core that has a transition decoder to predict the states from the GRU internal state. ADMv2 modifies this by adding a State encoder before the GRU, so that the state can be embedded beforehand into a better representation. This is trained end-to-end. Another addition is the PARoll prediction mechanism to make multiple forward predictions more practical compared to the method used in the original ADM. To give a bit of background: as ADM allows predicting for any number of steps forward (less than m steps forward), we can construct multiple different predictions for the next state, based on from how many steps backward we will aim to predict the next state. ADM uses these multiple predictions to form an uncertainty estimate of for the next state prediction akin to ensemble methods. However, the original ADM performed this ensemble-like prediction in an inefficient manner, so the current work developed a PARoll framework, that essentially keeps m separate prediction streams for the different forward prediction lengths and runs these in parallel.\n\nFor offline policy learning, the uncertainty estimate (specifically the variance) is added to the rewards in the q-updates as a penalty with $-\\beta$ multiplier. They also give some theoretical performance bound.\n\nExperimentally, the work evaluates on D4RL, NeoRL offline RL benchmarks, and on the DOPE off-policy evaluation benchmark. The method achieves new state of the art results on D4RL and NeoRL and also had the top results on DOPE. They achieve these results while doing full episode rollouts with their model (i.e. 1000 step rollouts, starting from sampled states from the buffer).\n\nIn addition, they tested points such as the model prediction accuracy (which improved), rollout throughput (which improved greatly compared to ADM, but was lower than the ensemble-based method), and also checked correlation between model errors and the uncertainty quantification (there was a good correlation for their method).", "questions": "One way to strengthen the paper would be to also evaluate on OGBench:\nhttps://arxiv.org/pdf/2410.20092\n\nRegarding multi-step predictions, there are also other earlier related works:\n\nLearning Awareness Models, Brandon Amos et al. (ICLR2018)\nhttps://arxiv.org/pdf/1804.06318\n\nLearning Latent Dynamics for Planning from Pixels, Hafner et al. (ICML2019)\nhttps://arxiv.org/pdf/1811.04551\n\nOn the other hand, you already cited the works of Asadi (2018). But I think it would be better to cite such a work in the introduction, rather than in the appendix.\n\nThe original ADM paper also evaluated such correlations, and in their case, the ensemble model correlations are better than the ones reported in this paper. Where did the discrepancy come from?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The empirical performance appears very strong.\n\n- The writing was good.\n\n- Many metrics were tested, e.g., accuracy, uncertainty quantification, multiple benchmarks.", "weaknesses": "- Novelty is that not high as it mostly an extension of ADMv1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wHmLEyusvZ", "reviewer": "Reviewer_ri32", "summary": "This paper proposes ADM-v2 (encoding the initial state (s_0) as a hidden state (h_0), combining any-step prediction with PARoll parallel roll-out, and using any-step uncertainty as a conservative penalty). The approach is direct and feasible in engineering, demonstrating significant advantages in full-horizon generation and evaluation of offline RL. The paper has practical value, but currently lacks several key ablation techniques, details, and more rigorous robustness experiments.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Focusing on error accumulation and parallelization bottlenecks in long horizon roll-out, which are core pain points in offline MBRL.\n2.Encoding (s_0) as (h_0) and removing backtracking, the parallelization approach (PARoll) is easily implemented in engineering.\n3.The any-step supervised design significantly improves sample utilization, which is a reasonable path to enhance long-horizon capabilities.\n4. Includes roll-out error curves, throughput (samples/sec), offline evaluation (DOPE), and downstream policy performance (D4RL/NeoRL), providing multi-dimensional evidence.\n5. Using any-step variance as a conservative penalty is practically meaningful for offline learning security.", "weaknesses": "1.The design of compressing (s_0) into (h_0) is highly dependent on encoder selection and capacity. The paper lacks sufficient ablation and quantitative analysis of this (e.g., the impact of different encoder capacities, aux losses, and skip connections).\n\n2.Relies on ensembles, but lacks horizon-wise calibration evaluation (the relationship between ensemble variance and true error should be shown more systematically); the computational/memory overhead and trade-offs of ensembles are not adequately discussed.\n\n3.The paper claims a significant increase in throughput, but lacks reproducible PARoll pseudocode, batch organization examples, and memory consumption analysis; throughput comparisons may also be affected by implementation optimizations (a fairer baseline setting explanation is needed).\n\n4.Most metrics are based on open-loop any-step evaluation, but the final policy will be implemented under closed-loop conditions; the paper should supplement closed-loop rollout results or use hybrid training (scheduled sampling / one-step + any-step) to validate stability.\n\n5.Lacks robustness testing for insufficient data coverage or policy distribution drift (e.g., perturbations to initial states or actions, domain-shift testing).\n\n6. The paper needs to provide a more complete list of hyperparameters (ensemble size, k-sampling strategy, encoder/GRU structure, number of training steps, seed), as well as a commitment to release the code/training script.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9XFliuBci2", "reviewer": "Reviewer_s9SG", "summary": "The paper proposes a modification of the Any-step Dynamics model by Lin et al. Contrary to prior work, the authors propose to remove the per-step conditioning of the hidden state on the initial state in the dynamics model roll-out. They show that this leads to empirically better performance than comparable baselines, and give some theoretical insight into the performance of k-step models.", "questions": "The theoretical performs bound indeed shows a tighter dependency of the error on the roll-out horizon by a factor of $(1-\\gamma^m)$. However, this assumes that the m-step error can be controlled to be the same value as the one step error. While this seems to empirically be true in the (very low noise) environments addressed here, I don't see a principled reason for why this should always be the case. Could the authors comment on this? Effectively, is there there a reason to believe that $max_k \\delta^k$ does not depend unfavourably on $k$. Basically, is it always reasonable to assume that we can get the k-step error as small as the 1-step error.\n\nWhile the authors propose full horizon rollouts, they do not test their model as a short horizon model. However, the baseline ADM model performs better as a short horizon than as a long horizon model. Does ADM-v2 actually provide a benefit from doing full horizon rollouts, or would the returns increase even more if it was used at a short horizon? This is a vital experiment that would benefit the narrative of the paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The modification the authors propose is a simple change to the prior work, ADM, removing a conditioning of the dynamics model latent state on the first state in the trajectory. To clarify: I count simple yet impact-full changes as a strength, I think it is a good idea to thoroughly analyze simple and clear design choices.\nExperimental evidence supports the benefit of this design decision.\nThe authors also provide theoretical insight into the benefit of k-step model training in model-based RL that may be of independent interest to the community (see notes below).", "weaknesses": "The theoretical contribution is somewhat disjoint from the modification presented here. While it is reasonable to prove that a k-step model can lead to better performance than a one-step model, this proof applies to both the prior work and this work. I think some more targeted analysis of the specific design decision proposed here would have made the paper much stronger. This weakness is currently my main reason for recommending rejection: I do not think that the theoretical contribution serves to justify the design decision proposed here.\n\nBuilding on this, I overall do not think the paper provides a strong enough intuition for why removing the conditioning on the state helps empirically. I think some more targeted investigation (theoretical or empirical) would greatly strengthen the paper. For example, the analyses presented in Figure 4 do not show ADM-v2 performing significantly better than ADM, yet it leads to farther higher reward. Therefore, investigating e.g. the generalization properties of the method in more detail seem highly relevant to clarify what the modification is doing to the community. Similarly, the analysis in Figure 5 is not tied to performance in the offline RL task, making it hard to assess the impact of an improved correlation.\n\nSome results do not align between prior work and this work. For example, the ADM paper reports a Correlation of 0.98 on the hopper-medium task, while this paper reports 0.871. The score reported in prior work outperforms the performance reported here, and I think it is important for the authors to comment on this. Is this explained by the horizon of the rollout?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)