Dynamics‑Aligned Diffusion Planning for Offline RL: A Unified Framework with Forward and Inverse Guidance
Authors: Zihao Wang
Ke Jiang
Xiaoyang Tan
Venue: Accepted by TMLR ()

Abstract: Diffusion-based planning has emerged as a powerful paradigm for offline reinforcement learning (RL). However, existing approaches often overlook the physical constraints imposed by real-world dynamics, resulting in dynamics inconsistency—a mismatch between diffusion-generated trajectories and those feasible under true environment transitions. To address this issue, we propose Dynamics-Aligned Diffusion Planning (DADP), a unified framework that explicitly enforces dynamics consistency during the diffusion denoising process. DADP offers two complementary variants: DADP-F (Forward), which employs a forward dynamics model to ensure state-level feasibility, and DADP-I (Inverse), which leverages an inverse dynamics model to enhance action-level executability. Both variants share a unified guidance formulation that integrates task return optimization and dynamics alignment through gradient-based updates. Experiments on state-based D4RL Maze2D and MuJoCo benchmarks demonstrate that DADP-F and DADP-I outperform state-of-the-art offline RL baselines, effectively reducing dynamics inconsistency and improving long-horizon robustness. This unifies diffusion-based planning with physically grounded dynamics modeling.

Published in Transactions on Machine Learning Research (02/2026)
Dynamics-Aligned Diffusion Planning for Offline RL: A Uni- fied Framework with Forward and Inverse Guidance
Zihao Wang"
2th. manggnnuaa edu. cn ke jaa_jonng Sonngnnnaa...mm edu.cn
Ke Jiang"
Xiaoyang Tan
2. Lon @naaa edu cn
College of Computer Science and Technology,
Nanjing University of Aeronautics and Astronautics,
MIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China
Reviewed on OpenReview: https: /ttp://pppeeeeeeee net/ forum? id- h3hG6BuqU2
Abstract
Diffusion-based planning has emerged as a powerful paradigm for offline reinforcement learn- ing (RL). However, existing approaches often overlook the physical constraints imposed by real-world dynamics, resulting in dynamics inconsistencya mismatch between diffusion- generated trajectorics and those fcasible under true environment transitions. To address this issue, we propose Aanamics-Aligneee Diffusion Planning (DADP), unified framework that explicitly enforces dynamics cons during the diffusion denoising process. DADP offers two complementary variants: DADP-F (Forward), which employs a forward dynamics model to ensure state-level feasibility, and DADP-I (Inverse), which leverages an inverse dynamics model to enhance action-level executability. Both variants share a unified guidance formu- lation that integrates task return optimization and dynamics alignment through gradient- based updates. Experiments on the slate-based D4RL Maze2D and MuJoCo benchmarks demonstrate that DADP-F and DADP-I outperform state-of- thettf-tt--rttt offline RL baselines, effectively reducing dynamics inconsistency and improving long-horizon robustness. This work unifies diffusion-based planning with physically grounded dynamics modeling.
1 Introduction
Reinforcement. learning (RL) (Sution, 2018: Silver et all 2017) has demonstrated remarkable potential in solving complex sequential decision-makin making problems. However, its application in critical domains such as medical development (Fatemi ct all, 2022), autonomous driving (Fang et all 2022), and robotics con- trol (Sinha_el all 2022) is significantly constrained by the impracticality of trial-and-error mechanism. This limitation has spurred the development of offline RL (Kumar all 2020; Jiang et all, 2023, Yu all, 2020), which focuses on learning effective policies directly from pre-collected datasets.
Since offline RL relies solely fixed, behavior- driven datasets, most algorithms adopt conservative objectives that favor in-distribution actions. While such strategies mitigate extrapolation errors, they also induce myopic policy optimization (Jiang et al, 2028), focusing excessively on short-term rewards while neglecting the long-term consequences of early decisions. This limited foresight makes it challenging to synthesize reliable multi-step action sequencesan ability essential for modern robotic and autonomous systems. The problem further exacerbated by perception noise, actuation uncertainty, and distributional shifts between the behavior policy and the learned policy, which lead to error accumulation and performance degradation over extended horizons.
Recent advances in diffusion- based planning have provided probabilistic perspective on long-horizo decision-making by casting it as inference over optimal trajectory distributions. Inspired by control-as-
"Equal contribution. i "Corresponding author.

---

Published in Transactions on Machine Learning Research (02/2026)
Planned Trajectory
S₂
S₂
S
Target
Agent
Dynamics Inconsistency
Si
S22
Sn
3z.
Shy
Planned Actions
Executed Trajectory
Figure 1: Illustration of dynamics inconsistency in diffusson-based planning. The diffusion model generates a planned trajectory (blue) toward the target through denoising in trajectory space. When executed in the real environment, the actual trajectory (darkgroen) deviates duc to dynamics inconsistencythe mismatch between predicted and true transitions. This phenomenon motivates the proposed Dynamics- Aligned Dif- fusion Planning (DADP), which enforces forward (state-level) or inverse (action-level) consistency during trajectory generation.
inference (Lexiin, 20T and diffusion generative models (Croitom of all 20023 Janner of all, 2022). these methods train diffusion models to transform Gaussian noise into complete trajectories through iterative denoising. By operating directly in trajectory space, diffusion planners can model long-range temporal dependencies without committing to step-wise policy generation.
Despile this progress, existing diffusion-based planners oflen fail to capture fine-grained local transition structure. Consequently, generated trajectories may exhibit global coherence yet violate local transition feasibility when executed in the real environment. We refer to this phenomenon as dynamics inconsisten- cysystematic mismatch between predicted and true transitions that accumulates over time (Figure M). A key reason is that diffusion sampling is typically guided by return-related objectives and behavioral priors, while local dynamics constraints are not explicitly penalized during denoising, making errors compound over long horizons. Such inconsistencies can severely undermine reliabilily and robustiness, particularly in complex continuous-contro domains.
To address this challenge, we propose Dynamics-Aligued Diffusion Planning (DADP), a unified frame- work that explicitly enforces alignment between dilfusion-generattee trajectories and environment dynamics. DADP augments the standard denoising process with a ferentiable physies-consistency constraint during reverse diffusion. At cach denoising step, the model jointly optimizes task reward and local dynamic fidelity, producing trajectories that are not only high-reward but also physically realizable and dynamically stable. Conceptually, DADP transforms diffusion-based planning from pure generative inference into physics-guided optimization, bridging the gap between probabilistic mod- cling and real-world feasibility. Moreover, this integration enables the planner to capture richer temporal correlations and more faithful motion continuity, providing a principled way couple learned dynamics with trajectory generation under unified probabilistic framework.
energy term that serves as dif-
To comprehensively address dynamics inconsistency, further decompose it into two complementary pects and propose two independent variants under unified framework. DADP-F (Forward) incorporates forward dynamics model to penalize physically invalid state transitions, ensuring that generated next states are smoothly reachable given the current states and actions within the learned dynamics manifold. In con- trast, DADP-1 (Inverse) employs an inverse dynamics model to correct infeasible or non-exeen non-executable actions during denoising, guarantocing that generated controls are both consistent and realizable for achieving the de- sired state transitions. Each variant independently enforces dynamics alignment from different perspectives: DADP-F focuses on maintaining valid and physically plausible state transitions, while DADP-I emphasizes
2

---

Published in Transactions on Machine Learning Research (02/2026)
the executability and stability of generated actions. Together, they provide complementary insights into improving physical consistency, robustness, and long-horizon stability in diffusion-based planning.
Our main contributions are summarized as follows:
Unified Framework. We propose Dynamics-Aligned Diffusion Planning (DADP), a unified frame- work that explicitly integrates environment dynamics into diffusion-based trajectory generation for offline reinforcement learning.
Dual Implementations.
We develop two independent variantsDADP-F (Forward) and DADP- I (Inversc)whied) enforce state-level and action-level consistency, respectively, addressing dynamics alignment from predictive and corrective perspectives.
3. Principled Formulation and Mcchanistic Analysis. Unlike houristic improvements, we derive a mathematically grounded guidance objective (Eqs. 14 14-16) that explicitly models dynamics inconsistency as an energy potential. We provide a mechanistic analysis demonstrating how this gradient field effectively projects diffusion trajectories onto the feasible dynamics manifold, thereby mitigating compounding errors.
Comprehensive Evaluation. Extensive experiments on D4RL Maze2D and MuJoCo benchmarks demonstrate that both variants consistently outperform state-of-the-art diffusion- and model- based offline RL methods in terms of planning accuracy and robustness.
In this work, we focus on state-based offline RL, where the agent has access to low-dimensional proprioceptive states (e.g, joint angles, velocities). Extending our framework to high-dimensional sensory inputs (e.g.. pixels) or partially observable settings remains an important direction for future research.
2 Related Works
Offline Reinforcement Learning. Offline reinforcement learning (Offline RL) aims to learn high- performing policies from static datasets while maintaining conservatism to mitigate distributional shift. Most approaches achieve this by imposing value or policy constraints that keep the learned policy within or close to the behavior distribution. Early value-based methods such Batch-Constrained Deep Q-learning (BCQ) (Fujimoto et all, 201H) regularize the Q-target using behavior-clipped actions. Conservative Q- Learning (CQL) (KKunar el all, 2120) and Implicit Q-Learning (IQL) (Kossrikoxx el all, 202T) further con- strain the learned value function through aedintage-weighteed or expectile regression objectives, effectively avoiding out-of-distribution (OOD) actions. Although these mothods alleviate distributional bias, they often exhibit myopic optimization lendencies, focusing on immediate rewards and struggling to produce reliable long-horizon behaviors in complex control tasks.
Model-Based Reinforcement Learning (MBRL). Model-based reinforcement learning improved sample efficiency by explicitly learning environment dynamics for planning policy optimization. Representative works such as MOPO eYu_et_al 2220) and MOReL (Kidambi et all, 202T) mitigated model bias by pe- nalizing uncertainty or terminating rollouts that deviated from the data distribution. COMBO (Lu et all, 2021) combines mosel-based and value- based regularization to enhance offline policy learning. Despite their benefits, these approaches fundamentally depend on model accuracy: even small prediction errors can accu- mulate over time, resulting in unstable rollouts and degraded performance high-dimensional continuous environments.
Diffusion-Based Planning for Reinforcement Learning. Recent advances employed diffusion models to directly generate trajectorics in offline reinforcement learning. Diffuser (Janner all, 2022) introduced a denoising diffusion probabilistic model conditioned on returns to synthesize entire staleaction sequences. Building on this idea, Decision Diffuser (Ajay et all 2022) incorporated conditional guidance for desired outcomes, while hicrarchical extensions such as HDMI (L of all 2023) enabled multi-level trajectory plan- ning. Adaptive diffusion- planning (Zhon et all, 21231) introduced uncertainty estimation for online re-planning. While these methods effectively capture global trajectory structures, they often overlook local
3

---

Published in Transactions on Machine Learning Research (02/2026)
physical consistency, resulting in trajectories that may be statistically valid yet dynamically infeasible when executed. Our proposed Dynamics-Aligned Diffusion Planning (DADP) explicitly addresses this issue by integrating forward and inverse dynamics constraints into the denoising process, ensuring that generated trajectorics remain both high-roward and physically realizable.
Dynamics-Aware and Constrained Diffusion. While standard diffusion planners generate trajectories in an open-loop fashion, recent works have sought to enforce dynamics constraints more explicitly. Methods such as Squality-Constrainee DiffusionkKurtz & Burdick (2025) and Constrained DiffusersZhang all (2045) typically employ optimization techniquessuch augmented Lagrangian methods or primal-dual updatesto treat dynamics as strict equality constraints, primarily targeting direct trajectory optimization or safe plan- ning tasks. Similarly, approaches like DPCCRomer el all (2024) integrate continuity constraints directly into the generation pipeline via projection to enable constraint -esttraint-aaisfyinng imitation learning. While effective in their respective domains, these "hard constraint" methods often require complex inner-loop optimizations or sensitive hyperparameter tuning (c.g., Lagrange multipliers) and can be computationally intensive. Fur- thermore, they lypically foeus on satisfying constraints within expert demonst rations or analytical models, whereas goal is to extract optimal policies from suboptimal offline data. In contrast, our DADP frame- work adopts a soft guidance approach via energy minimization. Instead of enforcing hard constraints that effectively project trajectories onto a potentially imperfect learned manifold, DADP leverages gradient-baser guidance from learned forward/ iiverse VAFs. This formulation allows for a more flexible trade-off between reward maximization and physical feasibility, acts as a regularizer rather than a strict bound, and integrates seamlessly into the standard diffusion sampling loop with minimal computational overhead.
3
Method
3.1
Problem Formulation
Offline reinforcement learning (Offline RL) aims learn optimal decision- decision-making policies purely from pre- collected datasets, without any further interaction with the environment. The environment is modeled as Markov Decision Process (MDP) (S, P,r,,, where S and A denote the state and action spaces, P the transition kernel, the reward function, y the discount factor, and po the initial state distribution. An agent follows a policy æ S + A to generate trajectories = (so, ag. $1, a1...., ST-1, 0T) of horizon T. The optimal policy can be interpreted as producing the first action ag according to an optimal trajectory dist ion
xt (a|s) = p" (r|00 = sao))
(1)
where p" (T| s) denotes the distribution over optimal trajectories starting from the initial state s. In the offline RL setting, the agent has access only to static dataset D with empirical trajectory distribution PD(T). The learning objective becomes to infer an optimal trajectory distribution from this fixed dataset, which can be formulated as:
(T( = arg max E Ero Po(T)] + (r)))))
(2)
where [(T) - ETD r(se,a) a)) denotes the cumulative return.
This dual-objective formulation jointly encourages distributional alignment with the offline dataset and re- ward maximization for optimal control. The likelihood term serves as an implicit regularizer, alleviating sparse or delayed reward issues that often arise in long-horizon planning. This formulation naturally lends itself to diffusion- generative modeling, where the goal is to approximate the optimal trajectory distri- bution through iterative denoising.
3.2 Diffusion- based Trajectory Modeling
Diffusson probabilistic models (Soh-Dickstonn et all 2016 are powerful generative frameworks that synthesize complex data by iteratively denoising samples through a Markovian process. In the trajectory generation setting, diffusion model learns to map Gaussian noise to complete trajectories via. reverse diffusion
4

---

Published in Transactions on Machine Learning Research (02/2026)
process. Specifically, the generative process is defined sequence of reverse transitions p(r(-1[7') that invert a forward diffusion process q(r'|r/=1), where the latter progressively corrupts clean trajectories by adding Gaussian noise at each step. The marginal data distribution modeled by the diffusion process can be written as:
p(70)=
p(r I
P( dr
33)
where p(r*) is a standard Gaussian prior and 70 denotes the clean trajectory. The reverse transitions are typically parameterized as Gaussian distributions:
p()""1/)) = (1 (e((") E'V no ((o((")) E))
where p(r', ;i) and E(i) represent the mean and covariance of the reverse process, learned by neural network conditioned on the noisy trajectory 7¹ and the diffusion step i. The forward process q(r")r²=3) is defined fixed Gaussian corruption kernel (Sohi-Dickstenn et al 2IT)).
To integrate cettssion-based generation into decision decision-making, we adopt the control-as-inference perspec- tive (Levind, ) which formulates optimal control as a probabilistic inference problem. Let O₂ denote a binary optimality variable at time 4, where p(O, 1) = exp(r(8t,a))), The optimal trajectory distribution can then be expressed as a posterior:
p³ (²(r)= = p(r Our 1) P(T) = 1/7).
(5)
This formulation encourages sampling trajectories that are both behaviorally plausible (high p(r)) and revard-optimal (high p(Oa.T= 1/7))
Following (Janner et al MOZ when the optimality likelihood p(Ourr 1/r) is smooth, the reverse diffusion transitions can be approximated by:
EV p(T'~1 Oo.T) R +V
(6)
where 14, Σ are parameters of the unconditional diffusion model, and Vn.J(u) represents reward-guided gradient that steers denoising toward higher-return trajectories. This perspective bridges generative mod- eling and reward optimization, forming the basis for our proposed dynamics-aligned diffusion framework introduced in the next section.
3.3 Quantifying Dynamics Inconsistency
While diffusion-basod planning can capture global trajectory structure, it may violate local transition fcasi- bility. We quantify dynamics inconsistency two contexts: (i) differentiable objectives used for guidance, and (ii) a simulator-based metric used for evaluation.
Differentiable Inconsistency for Guidance
We quantify dynamics inconsistency from two complementary perspectives using learned dynamics models. Although our dynamics models are implemented as VAEs that parameterize conditional distributions, we denote their mean predictors by My and M; for notational simplicity.
State-level inconsistency (forward). Let 8 = My( az) be the mean prediction of the learned forward dynamics model. We define
T-1
T-1
EF(r)= =
Σ
1811 8
Σ
|t11
(8,,
(7)
1=0
Action-level inconsistency (inverse). Let à, = Ms(si, s141) be the mean prediction of the learned inverse dynamics model. We define
T-1
T-1
E((T)
Σ
|1a 0z112
Σ
||,
Mc(sr, $1.
(8)
1=0
0=
5

---

Published in Transactions on Machine Learning Research (02/2026)
Both quantities are diffcrentiable and are incorporated into DADP guidance objectives to regularize dynamics during trajectory generation.
Simulator-Based Inconsistency for Evaluation
To assess physical realism in experiments (e.g., Figure 5F executed transitions in the ground-truth simulator:
we compute the deviation between planned and
Eoval (T)
Σ t=
I 1.+immimml2
(9)
where Itm is obtained by rolling out the planned actions in the simulator (e.g., MuJoCo). Note that Eeval is used only for analysis and is not available during inference.
3.4 Learning the Dynamics Models
To enable disferentiable and robust. consistency guidance, both the forward and inverse dynamics models our framework are implemented variational autoencoders (VAEs). A probabilistic latent-variable formu- lation offers two main advantages over deterministic regression:
(1) It better handles the multi-modality inherent in offline datasets compared to deterministic regression, preventing the model from learning an invalid "average" transition:
(2) learns a smooth, structured latent manifold that yields stable and informative gradicnts for con- sistency guidance.
While the VAE explicitly models variance, this work, we primarily utilize the learned mean prediction for guidance. By leveraging the structured latent space, ensure that the gradients remain well-behaved during optimization, focusing prediction quality rather than explicit uncertainty weighting.
Formally, given obscrved transitions (s,, 4,, St41), the encoder maps inputs to a latent representation 2s. while the decoder reconstructs the conditional distribution of the target variableeither the next state or the actiondepending on the model type. Assuming a standard Gaussian prior p(z) = (),, both models are trained by maximizing the evidence lower bound (ELBO):
LELOO (no(s - (x)(((
ll)))))
(10)
where (z, y) denotes the inputtarget pair and o, & are the paramcters of the encoder and decoder, respectively. We instantiale this structure in two complementary forms:
Forward dynamics model. The forward dynamics model parameterizes the conditional distribution of the next state given (s,, az): (0 st, a.) ((r Ey a.)).
(11)
For guidance, we use the decoder mean and denote it as deterministic predictor:
My (St, a:) (s,,a,)
(12)
In training, we minimize the mean-squared error of the mean prediction:
T-1
Cj Σ [M)(s 1
(13)
Inverse dynamics model. The inverse dynamics model parameterizes the conditional distribution of the action given (st, $141):
pala 81, 8c+1) M(ui(st, 8t11), Ei(0,, 8141))
(14)

---

Published in Transactions on Machine Learning Research (02/2026)
Similarly, we denote the decoder mean
Mf(si, (14,))((66)8444)) 8141)
(15)
This model captures multiple feasible controls for the same transition and provides robust feasibility correc- tion during denoising.
In practice, the forward and inverse dynamics models are used separately to support two DADP variants. DADP-F employs the forward model to guide trajectory generation toward state-level consistency, while DADP-I utilizes the inverse model to enforce action-level consistency. Each model acts as differentiable prior that regularizes the diffusion process, ensuring that generated trajectories remain consistent with the underlying environment dynamics.
3.5
Dynamics-Alignets Diffusion Planning (DADP)
The proposed Dynamics-Aligned Diffusion Planning (DADP) framework integrates learned dynamics models into the reverse diffusion process to mitigate dynamics inconsistency. The key idea is to introduce a dynamics- alignment constraint during denoising, ensuring that generated trajectories remain both reward-optimal and physically feasible throughout the diffusion process.
During planning, two pretrained models are employed: a diffusson model po(r) and dynamics model M, which can be either the forward model M or the inverse model M,, depending on the DADP variant. Trajectory sampling begins by initializing Tu from a standard Gaussian prior (N I), with the initial state fixed to the agents current observation. The reverse diffusion process iteratively denoises the trajectory over N steps, where each step is represented by pr(r At each iteration, the diffusion model outputs a mcan (T²) and covariance E' that govern the sampling distribution, cnsuring both stability and diversity during denoising.
To enforce local dynamics alignment, we compute a dynamics inconsistency score each iteration. Depend- ing the variant, the formulation is follows:
State=level alignment (DADP-F).
T-1
E (F))) Σ
nll Al| s441l22
(16)
Action-level alignment (DADP-I).
T-1
ET(r)= = Σ m (M1(8188441) arl1? 0=
(17)
where % is a weighting coefficient. In our implementation, we adopt a uniform weighting scheme by setting " = for all timesteps. Empirical results suggest that the smooth gradients provided by the VAF latent manifold allowed for robust guidance without the need for complex, uncertainty- based adaptive weighting schedules. Thus, prioritize algorithmic simplicity and stability, leaving adaptive weighting for future work.
At each denoising step, the trajectory is refined using gradient-based guidance that balances task rewards and dynamics consistency. The gradient of the augmented objective is compuled as
g (AJ BF.).
(18)
where J reprosents the cumulative reward defined in Eq. A À controls the reward weighting, and ,3 determines the strength of dynamics aligument. The next trajectory is then sampled
+-1 N( + E.
(19)
7

---

Published in Transactions on Machine Learning Research (02/2026)
where adjusts the inffuence of the guidance term while E' preserves sampling variance for exploration. To ensure task-specific feasibility, we apply constraints C during denoising:
C(r) {so Scurrent: $goal (if specificd)),
(20)
where Scurrent is the current state and 8gaal is optional target. These constraints ensure that generated trajectories start from valid states and, when applicable, terminate at goal positions.
In summary, DADP extends standard diffusion-based planning by incorporating explicit dynamics alignment. The forward variant (DADP-F) promotes predictive reachability through state-level consistency, while the inverse variant. (DADP-I) enforces action-level leasibility. Both independently enhance the physical validity and stability of long-horizen diffusion planning. The overall procedure is summarized in Algorithm
Algorithm Synamicc-AAlgneee Diffusion Planning (DADP) for Offline Trajectory Generation
Diffusion model μas Dynamics model M (either forward M, or inverse M,); Planning horizon T; Number of diffusion steps N; Scaling factors a, X, 3; Covariances ( Constraints C. Initialize noisy trajectory 7 ~N(,II); observe initial state Scurrent-
Input:
3: for i N,..., do
4: Compute denoised mean + (Aa(T))
5: Extract state action pairs {(8,, az S141 1 from
Compute cumulative reward J +- Eto r(tt, 0z).
7:
Compute dynamics inconsistency term:
0 MM 0e) - 8tt1l3,
for DADP-F for DADP-I
+
E
M 8(+1) acl??
Compute joint guidance gradient: 9 vn(AJ BE). Sample denoised trajectory: 7'-1 ~ N( + aE' E*). 10: Project -'- to satisfy constraints C. (e.g., So Scurrent, optional = -goal)
11: end for
12:
Output: Final planned trajectory 7".
5
a,2
a-1
a,
$,1
a.,
S1z
a
Dynamics Model (VAE)
dencising
Encoder
Decoder
Diffusion Model
Returns
St_2
az2
S..1
ax4
a,
Se1
a.,
ST1k
a
Figure 2: Overview of the proposed Dyngmics-Aligned Diffussonn Planning (DADP) framework. The dynam- ics model, implemented as a variational autoencoder (VAE), provides local physical consistency guidance during the diffusion- based trajectory denoising process. For the forward variant (DADP-F), the dynamics model prodicts the next state given (st, at), while for the inverse variant (DADP-I), infers the correspond- ing action from (st, 8.+1) The diffusion model then refines the trajectory by jointly optimizing for reward maximization and dynamics alignment.
8

---

Published in Transactions on Machine Learning Research (02/2026)
4 Experiments
We conduct extensive experiments to comprehensively evaluate the proposed Dynamics-Aligned Diffusion Planning (DADP) framework. The study is designed to address the following key research questions:
1. Can DADP effectively reduce dynamics inconsistency and mitigate error accumulation in long- horizon planning?
Do the two variants, DADP-F and DADP-I, enhance robustness and generalization across diverse continuous-control environments?
We evaluated our approach on two widely used D4RL benchmarks: Maze2D for oonghoorioon, sparse-reward navigation, and MuJoCo for contimuous robotic control. Our implementation built upon the Diffuser code- base, employing U-Net backbone for trajectory denoising. Both forward and inverse dynamics models are implemented as variational autocncoders (VAEs) with fully connected encoderdecoder architectures. All models are trained using the Adam optimizer, with uniform weighting (7: = 1). Hyperparameters a, X, and B are tuned per environment via validation to ensure stable convergence.
In the following sections, we first describe the experimental setup and baseline configurations, then present quantitative results on Maze2D and MuJoCo benchmarks, followed by an in-depth analysis of the effects of dynamics alignment in DADP-F and DADP-I.
4.1 Maze2D: Evaluating Long-Horizon Planning
We evaluate the proposed DADP framework on the D4RL Maze2D benchmark, a suite of long-horizon navigation tasks characterized by sparse rewards. Maze2D) comprises three configurationsumaze,, medium, and largeooreeponding to increasing maze complexity and planning horizons. The agent receives reward of 1 only upon reaching the goal, so higher cumulative returns indicate shorter completion times within the fixed horizon. Following standard evalnation protocols, we report normalized scores, where higher values reflect more efficient planning and better task performance.
We compare DADP against representative model-free and model-based offline reinforcement learning algorithms. Modcl-free basclines include Batch-Constrained Deep Q-Learning (BCQ) (Fujimoto
Baselines.
atall
2019).
Conservative Q-Learning (CQL) aau 2020), and Implicit Q-Learning (IQL) (Kostrikov
al
(IZII) Model-based methods include ROMI (Wang et al. 2021), the original Diffuser (Janner et all,
2022
and improved variants such as Diffuuse- HER, Diffuser-AM (Kim all, 2122)), We also compare against Hicrarchical Diffusson for Offline Decision Making (HDMI) (L 2022) and the contemporary dynamics-
constrained method, DPCC (Römer el all 2023). which enforces continuity via hard manifold projection (reproduced using the official implementation). Together, these baselines cover diverse paradigms of value regularization, model-based rollout, and diffusion-based planning.
Results. Table II reports normalized scores Maze2D and its multi-task variant Multi2D. DADP-F and DADP-1 consistently outperform all baselines across maze sizes and task settings. In the single-task scenario, DADP-F achieves an average normalized score of 137.8, surpassing Diffuser (119.5) and HDMI (123.5). DADP-I attains a similar score of 137.5, demonstrating higher robustness across random seeds. In the more challenging Maze2D-Large environment, DADP-F achieves 147.7representing a 20% improvement over Diffuuse (123.0)lighlighhinng the benefits of explicit dynamics alignment.
In the multi-tas' setting, where start and goal positions randomized, DADP-F and DADP-1 again yield the best overall performance, achieving averages of 144.2 and 145.9, respectively. Both variants generalize effectively to unseen configurations, while traditional diffusion-based planners show degraded performance under distributional shift.
Representative planning and exccution results Mazc2D-Laree are shown in Figure Sal The first row shows planned trajectories, while the second and third rows correspond to executions under Diffuser and DADP-I, respectively. Each column depicts an independent episode. Diffuser often deviates between planned and
9

---

Published in Transactions on Machine Learning Research (02/2026)
executed paths due to cumulative modeling errors, whereas DADP-I maintains close alignment, confirming the stability benefits of dynamics alignment.
Comparison with DPCC. We further evaluated the official DPCC implementation (Romer et all, 21244 on the Maze2D benchmark. As shown in Table I. DPCC exhibits significantly lower performance compared DADP (e.g., 50.2 vs. 130.2 in U-Maze). This performance gap primarily stems from DPCC's reliance on iterative optimization-aase
projection, which acts as hard constraint. On suboptimal offline datasets, this rigid projection can force trajectories into valid but low-reward regions of the dynamics manifold, limiting the planner's ability to stitch optimal paths. In contrast, DADP employs principled formulation based on soft, gradient-basod energy minimization. This approach graccfully balances reward maximization with physical feasibility, allowing the planner to effectively "stitch" high-reward segments while penalizing dynamics violations. The advantage evident in the complex Maze2D- Large task, where DADP-F achieves 147.7 compared DPCC's 96.5, highlighting the robustness of our framework.
Figure BE further provides a quantitative comparison of dynamics inconsistency (evaluated against the ground-trute toann simulator defined in Eq across representative episodes. Diffuser exhibits rapidly increasing inconsistency over time, indicating compounding errors, while DADP-1 maintains low deviation throughout execution. These findings confirm that inverse dynamics guidance effectively suppresses error propagation and improves physical consistency in long-horizon planning.
Table 1: Normalized scores on D4RL Maze2D and Multi2D benchmarks.
Rnvironment
BCQ
cqu
IQL.
ROMI
DD
HDMI
Diffoser
Diffower (HRR)
Diffuer (AM)
DPCC
DADP-P (Ours)
DADP-I (Ours)
Maseil U-Maze
47.4
189.5
120.1
113.9
125.4
121.4
130.2 0.
131.5 1
Maa:2D Medium
121.5
130.3
Maae2D Lorge
68.6
135.9
123.0
135.8
147.7
Stugle-taok Average
Mulu2D U-Mase
26.5
126.2
128.9
1827
190.8
MuluiD Modiam
12.1
120.7
127.2
130.7
Mulu2D Large
18.9
190.5
185.4
1893
158.1
158.7 ::
Multi-tosh tosk Average
16.9
129.5
132.8
129.4
185.0
139.6
91.9
144.2
146.9
Planning
Execution Reality
Diffuser
DADP-1
(a) Visualization of planned and executed trajectories on the Maze2D-Large benchmark. Each column corre- sponds an independent episode. The first shows trajectories planned by the Diffuser, while the second and third rows depict their actual executions under Dif- fuser and DADP-I, respectively. Compared to Diffuser, DADP-1 maintains closer alignment between planned and executed paths, indicating improved dynamics con- sistency.
(b) Dynamics inconsistency scores for the three episodes illustrated in (a). Each plot shows the deviation be- tween planned and executed trajectories for Diffuser and DADP-L Lower scores indicate higher fidelity and better execution consistency.
Figure 3: Comparison between trajectory visualization and dynamics consistency analysis on the Maze2D- Large benchnark. (a) visualizes planned and executed trajectories, while (b) reports corresponding dynamics inconsistency scores (measured as the Euclidean distance against the ground-truth simulator states defined in Eq. D) Lower scores indicate higher fidelity and better exccution consistency.
10

---

Published in Transactions on Machine Learning Research (02/2026)
4.2 MuJoCo: Robust Continuous Control
Datasets. We further evaluate DADP the DARL MuJoCo contintouus-ontroll benchmarks, including HalfChectah, Hopper, and Walker2d. Each environment is evaluated under three datasetsmedium--xxot medium, and medium-replaythat capture increasing degrees of data imperfection and distributional shift. Each dataset contains one million Iransitions generated by policies of different quality levels. The medium- expert dataset combines trajectories from expert and medium policies in equal proportions, the medium datasct derived from a partially trained Soft ActorCritic (SAC) policy, and the medium-replay datasct aggregates all replay bulfer transitions up to medium- peretuum--pformmneeee policy.
Baselines. We compare DADP with representative offline reinforcement learning algorithms from both model-free and model-based paradigms. Model-free basclines include Behavior Cloning (BC), Conservative Q-Learning (CQL) (Kumar et all, 2020). Implicit Q-Learning (IQL) (Kostrikow et all, 2121), and Decision Transformer (DT) (Chen et all 2021). For model-based approaches, we consider MOPO of 202() and
Diffuser (Janner et all, 2022). This comprehensive set of baselines enables a fair comparison across policy- learning, model-based, and diffusion- based paradigms, isolating the contribution of dynamics alignment improving robustness and stability.
Results. Table @ reports normalized scores across all MuJoCo environments and datasets. Both DADP-F and DADP-I show superior performance compared to most baseline methods, where DADP-I attains the highest average normalized score of 81.2 across all evaluated tasks. DADP-F excels on tasks with smoother and more prodictable dynamics, such as HalfChoctah and Walker2d, whereas DADP-I demonstrates superior robustness in stochastic envirouments like Hopper. Compared to Diffuser, both variants achieve substantial gainsparticularly on the medium and medium-replay datasetsindicating that explicit dynamics alignment effectively mitigates model bias and stabilizes long-horizon control. Overall, these results confirm that incorporating forward or inverse dynamics guidance significantly enhances the stability, robustness, and ion of diffusson-based offline reinforcement. learning.
Table 2: Performance comparison on the D4RL MuJoCo benchmark. Each entry reports normalized scores for different offline RL algorithms across three datasets (xeeitm-expert, medium, and medium-replay). Results for DADP-F and DADP-I averaged over 100 evaluation episodes and reported as mean é standard error. The highest score for each task is highlighted in bold.
Dataset
Enviromment
CQL
IQL
DT
MOPO
Diffuser
DADP-F (Ours)
DADP-I (Ours)
Medium-Exeee
HalfChectah
91.6
86.7
86.8
63.3
88.9
89.6 1: 0.1
92.3 +
Medium Mediun-Expert
Hopper
105.4
91.5
107.6
23.7
103.3
104.4 + 1.0
108.1 + 1.0
Medium-Exppe Expert
Walker2d
108.8
109.6
108.1
44.6
106.9
107.6 + 0.1
108.0 = 0.1
Medium
HallCheetah
44.0
47.4
42.6
42.3
44.9 + 0.1
43.9 0.1
Medium
Пopper
58.5
66.3
67.6
28.0
77.3 + 0.7
96.2 + 0.7
Medium
Walker2d
72.5
78.3
17.8
79.6
0.4
0.4
Medium-Replay
HalfCheetah
45.5
44.2
36.6
53.1
37.7
38.7 + 0.3
38.8 + 0.3
Replay
Hopper
95.0
94.7
82.7
67.5
93.6
95.2 + 0.4
95.6 + 0.5
Medium-Replay
Walker2d
77.2
73.9
66.6
39.0
70.6
77.9 + 2.4
71.9 + 1.8
Average
77.6
77.0
74.7
42.1
77.5
79.7
81.2
4.3 Ablation Study and Sensitivity Analysis
To assess the contribution of the dynaaicc-alignment mechanism, we conduct ablation experiments on both the Maze2D and MuJoCo benchmarks by removing the dynamics-consistenas term from DADP. As shown in Figure H. excluding this term (DADP w/o Alignment) leads to a consistent drop in normalized scores across all Maze2D variants under both single-task and multi-task settings. This confirms that the alignment component effectively mitigates long-horizon error accumulation and enforces physical feasibility during trajectory generation.
We further extend the ablation the MuJoCo control tasks (Table 3) where both DADP-F and DADP-I out- perform the unaligned variant across all datasets. DADP-F performs better in environments with smoother
11

---

Published in Transactions on Machine Learning Research (02/2026)
dynamics such HalfCheetah and Walker2d, while DADP-I exhibits higher robustness in stochastic and underactuated environments such as Hopper. These results demonstrate that dynamics alignment provides consistent benefits across diverse control regimes and varying environment complexities.
To analyze the robustness of DADP with respect to hyperparamcter choices, we perform sensitivity studies on the inverso-aligned variant (DADP-I) using the MuJoCo benchmark. Here, À controls the weighting of the reward guidance term, and adjusts the strength of the dlyamics-alignmeen constraint during denoising. The step-size coefficient is found to be relatively insensitive and is fixed to 1 in all experiments. Figure D presents the sensitivity curves for À (top) and 3 (bottom) across three environments: HalfCheetah, Hopper, and Walker2d. Performance remains stable over several orders of magnitude for both parameters, highlighting that DADP-I is robust to hyperparameter variations and requires minimal tuning effort.
Discussion on Combining Forward and Inverse Guidance. natural question arises: can we com- bine DADP-F and DADP-I to enforce both state and action consistency simultancously? While theorctically appealing, this combination introduces practical challenges. First, calculating gradients from two distinct dy- namics models doubles the computational overhead during inference. Second, jointly optimizing for reward, forward consistency, and inverse consistency requires balancing three weighting hyperparameters (A, 3r, B1), which can lead to gradient conflicts and increased tuning complexity. Empirically, we found that selecting the variant best suited for the task structure (e.g., DADP-T for complex control, DADP-F for navigation) yields significant gains while maintaining algorithmic simplicity. We leave the efficient unification these objectives to future work.
Ablation Study on Maze2D
Ablation Study on Multi2D
160
160
166
158
14/
129
140
120
140
136
130
131
128
127
1:37
121
123
120
100
U-Maze
Medium
Large
U-Maze
Medium
Large
DADP (w/o Alignment)
DADP-F
DADPP-1
DADP (w/o Alignment)
DADP-F
DDADP-1
(a)
(b)
Figure 4: Ablation study results on the Maze2D and MultiDD benchmarks. (a) Maze2D represents single- task planning with fixed start and goal locations, while (b) Multi2D evaluates the multi-task setting where both start and goal positions are randomly sampled across episodes. Each bar shows the normalized score of DADP variants with and without dynamics alignment.
Table 3: Ablation study results on the DARL MuJoCo benchmark. Each dataset group (Medium-Expent Medium, Beddimm-Replayy contains three continuous control environments: HalfCheetah, Hopper, and Walker2d.
Medium- Mediurn-Expert
Medium
Medium-Replan Replay
Method
HallChoetah
Hopper
Walker2d
HalfCheetah
Hopper
Walker2d
HalfCheetah
Hopper
Walkerdd
DADP (w//o Aligument)
88.9
103.3
106.9
42.8
74.3
79.6
37.7
93.6
70.6
DADF-F
101.4
107.6
38.7
DADP-I
92.3
108.1
108.0
43.9
96.2
80.5
38.8
95.6
71.9
4.4 Open-Loop Multi- Multi-Step Execution Analysis
Most previous diffusion-bante planners adopt closed-loop execution scheme, where the agent replans after every action using the most recent observation. While this strategy effectively limits error accumulation, it
12

---

Published in Transactions on Machine Learning Research (02/2026)
HalfCheetah
Hopper
Walker2d
45
100
Nemalize Score
42
Normalized score
90
Normalized Score
80
39
80
70
36
70
65
80
50
1E-07
15-05
1E-03
1E-01
35-07
1E-03
1E-01
HalfCheetah
Hopper
Walker2d
42
105
80
Nomalize Store
39
Normatzed Store
95
Nomallze Score
75
70
36
85
65
33
75
60
30
65
55
16-07
1E-05
1E-03
1E-01
1E-07
1E-05
1E-03
1E-01
1E-07
1E-05
1E-03
1E-01
Figure 5: Sensitivity analysis of hyperparameters À and 8 on the MuJoCo benchmark. The top figure reports results on the Medium setting, showing the effect of X, which controls the weighting the reward objective. The bottom figure shows the Medium-Replay setting, illustrating the influence of B, the dynamics alignment weight. Both Ả and are plotted logarithmic scale.
incurs substantial computational overhead. In practical scenarios, however, reducing planning frequency by executing multiple actions per planan open-loop strategycan greatly improve efficiency.
Figure illustrates the trade-off between average rollout time and performance for DADP-1 on the MuJoCo Hopper-medium task. As the number of executed actions per plan increases, the overall computation time decreases substantially, with only marginal degradation in normalized performance.
These results demonstrate that DADP-I maintains strong robustness under reduced planning frequency, achieving an effective balance between compulat ional efficiency and control performancean essential property for real-time deployment of diffusion- based planners.
Trade-off between Rollout Time and Performance
Performance (Normalized Score)
60
75
65-
60
DADR-I
1000
1500
2000
2500
Average Rollout Time (s)
Figure 6: Trade-off between rollout time and performance on the MuJoCo Hopper-medium task. Each point represents the average performance of DADP-I under different open-loop execution lengths. The results show that DADP-I maintains stable performance as rollout time decreases, achieving an effective balance between computational efficiency and control accuracy.
13

---

Published in Transactions on Machine Learning Research (02/2026)
5 Limitations and Future Work
Despite its promising results, DADP faces several limitations. First, the iterative inference of diffusion models incurs high computational costs, limiting scalability to real-time applications. Second, while we employ probabilistic VAEs to capture dynamics, our current guidance mechanism relies on the predicted mean with a. fixed weight (7: = 1). This formulation does not fully exploit the model's uncertainty estimates, potentially introducing model bias under severe distribution shifts. Consequently, the planning performance is effectively upper-bounded by the fidelity of the learned dynamics. Incorporating adaptive weighting based on VAE variance represents promising avenue to further enhance safety in uncertain regions. Finally, DADP assumes static offline datasets without online adaptation. Furthermore, our current evaluation is limited to fully observable, state-based continuous control tasks. Applying DADP to high-dimensional pixel-based observations, partially observable MDPs, or contact-rich manipulation tasks would likely require integrating latent dynamics models or world models, which we identify as a key area for future work.
6 Conclusion
This paper presented Dynamics-Aligned Diffusion Planning (DADP), a unified framework that integrates explicit dynamics modeling into diffusion-basee trajectory generation to address dynamics incousistency in offline reinforcement learning. By incorporating forward and inverse dynamics guidance, DADP enforces physical feasibility during denoising and enhances stability in long-horizon planning. Extensive experiments on the D4RL Maze2D and MuJoCo benchmarks demonstrated that DADP significantly improved planning accuracy, robustness, and generalization over prior diffusion- methods.
Broader Impact Statement
This work introduces Dynamics-Aligned Diffusion Planning (DADP) to improve the reliability of offline rein- forcement learning in complex control tasks. While our method demonstrates enhanced planning consistency, we acknowledge potential risks and societal impacts associated with its deployment in real-world scenarios.
Safety in Critical Applications. Our paper motivates the use of DADP in domains such as autonomous driving and medical treatment. However, the framework relies on learned dynamics models (VAEs) to enforce consistency. In safety-critical settings, learned models may suffer from hallucination or prediction errors, particularly when the agent encounters dusti--distribution (OOD) states not well-represented in the offline dataset. Relying solely these Iearned priors without rigorous uncertainty quantification hard safety constraints (e-g,, rule-based shields) could lead to dangerous failures. Practitioners should exercise caution and consider integrating DADP with verifiable safety protocols before real-world deployment.
Computational Cost and Environmental Impact. Diffusion-based planning involves iterative denoising processes, which incur significantly higher computational costs and inference latency compared to standard MT.P-basod policies (o.g., CQL. IQL.). This incrcased computational burden translates higher energy consumption and a larger carbon footprint, especially during large-scale training and deployment. Future research should focus improving the sampling efficiency of DADP (e.g., via distillation or fast samplers) to mitigate its environmental impact and enable deployment on resource-constrraiee edge devices.
Acknowledgments
This work is partially supported by National Natural Science Foundation of China (62476128).
References
Anurag Ajay, Yihu Du Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is condi- tional generative modeling all you need for decision-making? Xiv preprint arXiv:221 15657, 2022.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084 15097, 2021.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10650-1866) 2023.
Xing Fang, Qichao Zhang, Yinfeng Gao, and Dongbin Zhao. Offline reinforcement learning for autonomous driving with real world driving data. In 2022 IEEE 25th International Conference Intelligent Trans- portation Systems (ITSC), pp. 3417-3422. IEEE, 2022.
Mehdi Fatemi, Mary Wu, Jeremy Petch, Walter Nelson, Stuart J Conmolly, Alexander Benz, Authony Car- nicelli, and Marzych Ghassemi. Semi-markov offline reinforcement learning for healthcare. In Conference on Health, Inference, and Learning, pp. 119 137. PMLR, 2022.
Justin Fu, Aviral Kumar, Ofr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. arXiv preprint (XXu:2004002299, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052 2062. PMLR, 2019.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902 9915, 2022.
Ke Jiang, Jia- Yao, and Xiaoyang Tan. Recovering from out-of-sample states via inverse dynamics in offline reinforcement learning. Advances in Neural Information Processing Systems, 36:38815-38522 2023.
Rahul Kidambi, Aravind Rajeswaran, Pranecth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810-21823, 2020.
Sungyoon Kim, Yunscon Choi, Daiki E Matsunaga, and Kee-Eung Kim. Stitching sub-trajectories with conditional diffusson model for goal-conditioned offline rl. In Proceedlings of the AAAI Conference Artificial Intelligence, volume 38, pp. 18160-13161 2024.
Ily Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning.
arXiv preprint ar arXi::2110.061 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offine reinforce- ment learning. Advances in Newral Information Processing Systems, 33:1179-1191, 2020.
Vince Kurtz and Joel W Burdick. Equality constrained diffusion for direct trajectory optimizatiin. In 2025 American Control Conference (ACC), pp. 535 540. IEEE, 2025.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint Xiv::1180..0 0009,, 2018
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pp.
PMLR, 2023.
Diganta Misra. Mish: A self regularized non-monotonic activation function. orXie preprint arxi:: net::198800888 2019.
Ralf Römer, Alexander von Rohr, and Angela P Schoollig. Diffusion predictive control with constraints. arXiv preprint xrXi::2412.093/2, 2024.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonogiou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without luman knowledge. nature, 550(7676) 359, 2017.
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. Sírl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Conference Robot Learning, pp. 907-917. PMLR, 2022.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibriun thermodynamics. In International conference on machine learning, pp- 2256-2226222222 2265. PMLR, 2015.
Richard S Sutton. Reinforcement learning: An introduction. Bradford Book, 2018
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline re- inforcement learning with reverse model-based imagination. Advances in Newral Information Processing Systems, 34:29420 29432, 2021.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3 3-19, 2018.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing 2020.
Systems,
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offiine model-bace based policy optimization. Advances in neural information processing systems, 34:28954 28967, 2021.
Jichen Zhang, Liqun Zhao, Antonis Papachristoddouln, and Jack Umenberger. Constrained diffusers for safe planning and control. aixie preprint urXiv:2506. 12544, 2025.
Siyuan Zhou, Yilun Du, Shun Zhang, Mengdi Xu, Yikang Shen, Wei Xiao, Dit- Yan Young, and Chuang Gan. Adaptive online replanning with dilfusion models. Advances in Newral Information Processing Syslems, 36, 2024.
A Dataset Details
(a) Maze
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 7: (a) Maze: A point-mass (green dot) agent needs to plan path from the starting position to reach the target location (red dot). (b) HalfCheetah: A two-dimensional cheetah robot that controls its joints to run as fast as possible to the right. (c) Hopper: A two-dimensional monopod robot where the agent needs to control its hinge to perform rightward hopping motion. (d) Walker2d: Based on Hopper, an additional leg is added, requiring the agent coordinate the movements both legs to achieve fast walking while maintaining balance.
A.1 Maze2D
The Maze2D benchmark in D4RL (L el all 2020) is designed for evaluating long-horizon navigation and sparse-reward planning. It consists of three environments: U-Maze, Medium, and Large, which progres- sively increase complexity and trajectory length. Each datasct contains trajectorics collected by a goal- conditioned controller navigating through 2D mazes of varying sizes. The agents state consists of the (x, y) location and velocities. A sparse reward of +1 is provided only upon reaching the goal, while intermediate
16

---

Published in Transactions on Machine Learning Research (02/2026)
steps receive zero reward. Following the standard D4RL evaluation protocol, compute normalized scores by first normalizing the cumulative episode rewards between the minimum and maximum returns of refer- ence (random and expert) policies, and then scaling the result by 100. Thus, score of 100 corresponds to the expert-level performance, while represents random behavior. The final score is obtained by averaging normalized returns over all evaluation episodes.
A.2 MuJoCo
The MuJoCo benchmark D4RL contains three standard continuous control tasks: HalfCheetah, Hop- per, and Walker2d. Each task is evaluated on three offline datasetsmedium-eepprt medium, and medium- replayeach containing one million transitions generated by Soft Actor-Critic (SAC) agents trained to different degrees of proficiency.
Medium-Expent 50% expert and 50% medium-level trajectories mixed uniformly.
Medium: trajectorios from a partially trained SAC policy paused at mid-training.
Medium-Roplay: all transitions from the replay buffer during SAC training up to the medium performance level.
All enviromments provide continuous state and action spaces with dense rewards defined by task-specific phys- ical performance metrics (e.g,, forward velocity, energy cost, stability). We follow the D4RL normalization protocol, where a score of 0 corresponds to a random policy and 100 to an expert policy.
A.3 Data Usage Protocol
All experiments strictly follow the experimental setup of Diffuser (Janner et all, 2022). Datasets are used exactly as provided by D4RL without any additional filtering, augmentation, or relabeling. We ensure fair comparison by adopting identical training and evaluation splits across all methods. To ensure statistical re- liability, each experiment is performed using independent training seeds. Unlike deterministic policies, our diffusion-basod planner is stochastic, requiring extensive evaluation rollouts. Consequently, the reported results correspond to the aggregate of 150 evaluation episodes for Maze2D (3 seeds x 50 rollouts) and 100 evaluation episodes for MuJoCo, strictly following the standard evaluation protocols of the baselines. All reported results include the mean and standard error across the training seeds.
B Model and Network Architecture Details
B.1
Diffusion Model
The diffusion model used in DADP follows the same architecture as the original Diffuser (Janner of all, 2022). adopts a one-dimensional temporal U-Net composed of six repeated residual blocks, designed capture hierarchical temporal dependencies within trajectory sequences. Each residual block contains two temporal convolution layers, each followed by group normalization (Wi Hd, 2018) and Mish activation function (Misra, 2UT)). Timestep embeddings are generated by single fully connected layer and are added to the activation of the first temporal convolution within each block to provide diffusion-step awareness. This structure enables stable denoising in the trajectory space while maintaining temporal smoothness and inter-step consistency.
B.2 Return Predictor Architecture
The return predictor J(T) utilizes the same temporal U-Net encoder backbone as the diflusion model. This shared architecture ensures feature compatibility and representational consistency. Structurally, the return predictor retains the encoder layers but replaces the decoder with a final Multi-Layer Perceptron (MLP) projection head, This head maps the aggregated trajectory embedding to a scalar value representing the predicted cumulative return.
17

---

Published in Transactions on Machine Learning Research (02/2026)
B.3 Dynamics Model
The dynamics model, used for both forward and inverse consistency guidance, is implemented a varia- tional autoencoder (VAE). Each VAE consists of a three-layer fully connected encoder and decoder, each with a hidden dimension of 256 and RcLU activations. The encoder outputs the mean and log standard deviation of a 32-dimensional latent variable 2, sampled via the reparameterization trick. The latent variable captures stochasticity and multi-modality in the environment dynamics, enabling smooth and differentiable consistency gradients.
C Training Configuration
C.1 Optimization and Training Setup
All models are trained using the Adam optimizer with default momentum parameters. For the diffusion model, we use a learning rate of x 10 and batch size of 32. For the dynamics models (both forward and inverse), the learning rate set to 10-3 with a batch size of 64.
All models are trained using the same datasets as those used for the diffusion process. Each task (Maze2D MuJoCo) is trained on its corresponding D4RL dataset independently, ensuring that the diffusion and dynamics models are both exposed to consistent trajectory distributions. This aligmment between training datasets guarantees that the dynamics priors provide coherent and task-specific consistency guidance during planning.
C.2 Diffusion and Return Predictor Training
The diffusion model is trained following the standard denoising diffusion probabilistic modeling (DDPM) objective:
Lai = E,... [l]c - co( 1)]))
(21)
where Eo denotes the noise prediction network parameterized by the temporal U-Net. We set the diffusion
horizon N 20 in MuJoCo and N 256 in Maze2D.
The return predictor is trained via supervised regression minimize the Mean Squared Error (MSE) between the predicted and actual discounted cumulative returns J(T) "fr="r(ss, az). To ensure training stability and consistent. gradient scaling, we apply GaussianNormalliannn mean, unit variance) based on statistics computed from the training dataset. We use the exact same training and validation splits as the diffusion model to prevent data leakage.
to standardize the target returns (zero
C.3 Dynamics Model Training
Both the forward and inverse dynamics models are trained independently using the variational autoencoder (VAE) objective. Each model employs a latent dimension of 32, with both encoder and decoder composed of three fully connected layers of size 256, each followed by ReLU activation. The encoder outputs the mcan and log standard deviation of the latent variable, which is sampled via the reparamcterization trick for differentiability.
Training is conducted using the Adam optimizer with learning rate of 1 x 10 and batch size of 64. Each model is trained for 3K epochs using the same datasct as the diffusion model, ensuring consistent trajectory distributions across components. Model checkpoints are saved periodically (every 1000 epochs) and the best model is selected based on validation loss. All training runs employ the same data preprocessing and normalization pipeline as in the diffusion model, with no data augmentation or relabeling applied.
C.4 Hyperparameter Settings
During planning, the key guidance coefficients are selected per environment via validation: 1.0 (stop size), X E [0.8, 1.0] (reward weight), and e [10-?, 2 10"] (dynamics alignment weight). These ranges balance reward optimization and dynamics consistency.
18

---

Published in Transactions on Machine Learning Research (02/2026)
C.5 Hardware and Software Configurations
All experiments are conducted workstation equipped with two Intel Xeon Gold 5218 CPUs (2.30GHz, 32 cores in total), four NVIDIA GeForce RTX 2080 Ti GPUs (1IGB cach), and 256GB of system memory. The operating system is Ubuntu Linux with CUDA 13.0 and driver version 580.95.05.
D Dynamics Model Accuracy
To ensure the reliability of the dynamics guidance, we evaluated the prediction accuracy of both the forward and inverse dynamics models on held-out test set (10% of the dataset). Table ED reports the Mean Squared Error (MSE) for state prediction (Forward) and action reconstruction (Inverse). Note that the absolute mag- nitude of MSE varies across environments due to differences in state dimensionality and feature scales (e.g.. HalfCheetah involves higher-velocity state components compared to Maze2D). Overall, the consistent low test errors relative to the scale of each environment confirm that the learned models capture the underlying dynamics effectively and provide accurate gradients for trajectory refinement.
Table 4: Held-out Test Mean Squared Error (MSE) for the learned Forward and Inverse dynamics models. The results demonstrate that the models achieve reasonable prediction accuracy across diverse datasets, ensuring reliable gradient guidance for the diffusion planner.
Environment
Forward Model Test MSE
Inverse Model Test MSE
MuJoCo Benchmenks
HalfCheetah Medium-Expert
0.8538
0.0086
HalfCheetah Medium
0.9465
0.0089
HalfCheetah Reeli
1.1733
0.0148
Hopper Oodium-Expert"
0.0360
0.0320
Hopper Medium
0.0373
0.0254
Hopper Medium-Replay
0.0762
0.0486
Walker2d Medium-Expee Exppert
0.3338
0.0322
Walker2d Medium
0.3498
0.0320
Walker2d Medium-Replay
0.8267
0.0879
Maze2D Benchmarks
Maze2D U-Maze
0.0014
0.0183
Maze2D Medium
0.0022
0,0150
Maze2D Large
0.0016
0.0204
E Efficiency and Latency Analysis
E.1
Sampling Steps vs. Execution Horizon
Sampling Steps (N) Constraints. A potential strategy to accelerate inference is reducing the number of diffuuson denoising steps N. However, in our implementation, the diffusson model is trained with a fixed noise schodule corresponding to N steps (e.g., = 20 for MuJoCo). Unlike consistency models or DDIM, directly reducing N during inference without retraining typically leads to a misalignment between the reverse process and the learned dynamics, resulting in dynamically infcasible trajectorics. Since our current N is already relatively compact to ensure fast planning, further reducing yields diminishing returns in latency while severely compromising performance.
Exccution Horizon Tradc-off. Instead, we identify Open-Loop Multi-Step Exccution (as analyzed in Section 4.4) as a more effective lever for optimizing the elficiency-accuracy trade-ofT. By executing a sequence of actions from a single planned trajectory before replanning, we amortize the inference cost of the diffusion model over l steps. As shown Figure E. this strategy significantly reduces the effective latency per step with minimal impact on control performance, providing a practical solution for real-time deployment.
19

---

Published in Transactions on Machine Learning Research (02/2026)
E.2 Wall- Clock Inference Latency
To explicitly quantify the computational overhead introduced by the dynamics alignment process, measure the average wall-clock time required to generate a single plan (batch size = 1) an NVIDIA GeForce RTX 2080 Ti GPU. Table details the inforence latency across three MuJoCo environments. Since DADP requires backpropagating gradients through the dynamics VAE at each denoising step, it incurs a slight additional computational cost. However, as shown the table, the relative slowdown is consistent and marginal (+s 1.05x) across different tasks. Combined with the open-loop execution strategy discussed above, this minimal overhead ensures that DADP remains highly efficient for practical deployment.
Table 5: Comparison of average wall-clock inference latency per plan (in seconds). Results are averaged over 100 independent planning steps single NVIDIA GeForce RTX 2080 Ti GPU. The results demonstrate that DADP introduces consistent and minimal latency overhead (~ 5%) compared to the baseline Diffuser.
Environment:
Diffuser (Baseline)
DADP-F (Ours)
DADP-I (Ours)
HalfCheetah- Medium- Medlau-Reppayy
1.16 + 0.12
1.22 + 0.03
1.22 + 0.08
Hopper-Medt Modium-Reepay
1.81 + 0.03
1.92 1 0.10
1.88 + 0.13
Walker2d- Walker2d-Medunn Saddumm-Repaay
1.77 1 0.12
1.80 1 0.18
1.87 1 0.14
Average Slowdown
1.00x
1.04x
1.05x
20


Reviews:
[{"review_id": "rXoRYsiCPx", "reviewer": "Reviewer_zZvq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\n\nThis paper investigates diffusion-based planning for offline reinforcement learning, highlighting a key failure mode: trajectories generated by a diffusion model can yield high rewards under a learned reward model but may be inconsistent with the true environment dynamics, resulting in significant discrepancies between planned and executed rollouts. To address this, the authors propose Dynamics-Aligned Diffusion Planning (DADP), which augments standard reward-guided diffusion planning with an explicit dynamics-alignment energy term derived from either a forward or inverse dynamics model trained from offline data. At each denoising step, the trajectory is updated using the gradient of a joint objective that increases predicted return while penalizing discrepancies between predicted and dynamics-consistent transitions. The method is instantiated in two variants: DADP-F (forward dynamics alignment) and DADP-I (inverse dynamics alignment), both of which are implemented via VAE-based dynamics models. Experiments on the D4RL Maze2D and MuJoCo benchmarks demonstrate that DADP enhances both planning performance and robustness to open-loop execution, outperforming prior diffusion-based planners, such as Diffuser and HDMI, and competing with strong offline RL baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The central empirical claim that incorporating explicit dynamics alignment into reward-guided diffusion planning improves performance and open-loop robustness relative to reward-only diffusion guidance is well supported by the reported results on D4RL Maze2D and MuJoCo benchmarks, including baseline comparisons, ablations, and sensitivity analyses. The paper provides evidence that DADP reduces a carefully defined measure of dynamics inconsistency and that this reduction correlates with improved return under open-loop execution in the tested environments. However, some of the broader claims about the general mitigation of compounding errors and their applicability to more complex domains are currently supported only indirectly by these experiments and heuristic arguments; as such, they should be interpreted as promising indications rather than fully general guarantees. Within the scope of the benchmarks and settings actually evaluated, the evidence is accurate and clearly presented.", "audience_yn": "Yes", "audience_explanation": "TMLR’s audience includes researchers in reinforcement learning, generative modeling, and decision-making under uncertainty, all of whom are likely to be following the recent trend of using diffusion models for planning and control.", "requested_changes": "- **Clarify the \"Theoretical Insight\" Contribution:** In Section 1 (Introduction), contribution #3 claims \"We provide an analysis showing that explicit dynamics alignment mitigates compounding errors...\" However, the paper does not appear to contain a formal theoretical analysis or derivation bounding the compounding errors. Section 3 presents the formulation of the method and metrics, but not a theoretical proof or a rigorous analytical bound. Please either add a formal theoretical analysis (e.g., a lemma bounding the divergence between planned and executed trajectories given the alignment term) or rephrase the contribution claim to reflect that the insight is empirical or architectural (e.g., \"We propose a formulation that explicitly targets dynamics consistency...\"). Mischaracterizing a heuristic formulation as \"theoretical analysis\" should be corrected.\n        \n- **Computational Cost Quantification:** The method requires computing gradients through a VAE (forward or inverse) at _every_ denoising step. This likely adds significant wall-clock overhead compared to the base Diffuser model. While Figure 6 discusses rollout time versus performance for open-loop settings, there is no direct comparison of inference latency per planning step between Diffuser and DADP. Please provide a table or text report detailing the average wall-clock time required to generate a single plan (or a batch of plans) for Diffuser, DADP-F, and DADP-I on the same hardware.\n\n- **Dynamics Model Quality Analysis:** The success of DADP relies on the accuracy of the learned dynamics VAEs. If the dynamics model is inaccurate, the alignment term would guide the planner away from feasible regions. Please include a brief evaluation of the dynamics models themselves. For example, report the reconstruction error (MSE) of the Forward/Inverse VAEs on a held-out test set for the different datasets. This will help clarify if the performance gains correlate with the quality of the learned dynamics.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aQ1NiiFUjn", "reviewer": "Reviewer_pnnn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Dynamics-Aligned Diffusion Planning (DADP) for offline RL: during each reverse-diffusion step, the denoising mean is guided by (i) a return objective and (ii) a dynamics-consistency objective so that sampled trajectories are both high-return and physically feasible. Two complementary variants are proposed: DADP-F uses a learned forward dynamics model to enforce state reachability; DADP-I uses a learned inverse dynamics model to enforce action executability. A unified gradient-guidance update and an explicit algorithm are provided.\n\nOn D4RL Maze2D/Multi-task and MuJoCo benchmarks, both variants outperform diffusion-based and standard offline RL baselines. Ablations show that removing dynamics alignment hurts performance; sensitivity analyses suggest robustness to guidance weights; an open-loop multi-step execution study illustrates a clear latency–performance trade-off.\n\n**strengths**\n\n1. Targets a real failure mode of trajectory diffusion (plan/execute mismatch) via differentiable dynamics alignment inside the sampler.\n\n2. Clean, unified formulation (return + consistency in one step) with implementation details that appear reproducible.\n\n3. Broad empirical support within D4RL, with ablations and sensitivity plots.\n\n**weaknesses**\n\n1. Relies on learned dynamics; under distribution shift, model bias can steer sampling toward infeasible regions.\n\n2. Training/inference cost is non-trivial; evaluation is mostly simulator-based (no pixel observations, partial observability, or contact-rich tasks).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper backs its central claims with competitive scores on multiple D4RL tasks, clear ablations (showing the benefit of alignment), and sensitivity/latency studies. The method is mathematically specified and algorithmically explicit.", "audience_yn": "Yes", "audience_explanation": "Readers in offline RL, diffusion-based planning, and guided generative sampling will find the “return + dynamics” in-sampler guidance useful. Practitioners concerned with planning latency will also be interested in the open-loop execution results and the practical guidance on when to prefer forward vs. inverse alignment.", "requested_changes": "**Critical**\n\n1. Current evidence is on state-based, simulator-style D4RL settings. Author can add some sentences explicitly limiting claims to state-based offline RL and outlining broader settings (pixels/POMDP/contact-rich tasks) as future work.\n\n**Nice-to-have**\n\n2. A small efficiency–accuracy study makes the method easier to adopt. Can author provide a small study with reduced sampling steps to chart accuracy–latency trade-offs?", "broader_impact_concerns": "We do not foresee specific ethical implications beyond standard considerations for offline RL research.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R5cfTNb6d5", "reviewer": "Reviewer_rjDr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary\n\nThe paper proposes Dynamics-Aligned Diffusion Planning (DADP), a trajectory diffusion framework for offline RL that explicitly enforces consistency with environment dynamics during reverse denoising. Two instantiations are presented: DADP-F uses a learned forward model to penalize state-level transition inconsistency, while DADP-I uses a learned inverse model to penalize action-level infeasibility; both are integrated via a unified guidance term that balances reward and dynamics-alignment gradients. Experiments on D4RL Maze2D and MuJoCo show that both variants outperform diffusion-based and standard offline RL baselines, with ablations and a planning-frequency study indicating improved long-horizon robustness and computational trade-offs.\n\n# Strengths\n\n- The paper addresses a concrete and recurring failure mode in diffusion planning, namely denoised trajectories that look globally coherent but violate local dynamics, hence leading to compounding error over horizons.\n- The approach is simple to implement on top of existing denoisers and thus potentially impactful for the growing body of diffusion-based decision-making.\n- The paper introduces a unified, energy-guided trajectory denoising scheme that incorporates differentiable dynamics consistency, with complementary forward (state-level) and inverse (action-level) variants.\n- The paper employs VAE-based forward/inverse models to provide smooth, stochastic guidance signals; this is pragmatic and plausibly more robust than purely deterministic dynamics regressors.\n- A Clean guidance formulation $g = \\nabla \\mu (\\lambda J − \\beta E)$ that directly augments the reverse diffusion mean, compatible with existing Diffuser-style planners.\n- The paper presents a broad evaluation on D4RL Maze2D and MuJoCo across multiple datasets, with consistent improvements over Diffuser and strong offline RL baselines. Useful ablations (w/o alignment) and sensitivity studies for $\\lambda$ and $\\beta$; open-loop multi-step execution analysis addresses a practical compute-performance trade-off.\n- A qualitative and quantitative analysis of “dynamics inconsistency” (planned vs executed trajectory divergence) strengthens the motivation.\n\n# Weaknesses\n\n- The claimed “theoretical insight” that alignment mitigates compounding error is not substantiated with formal results or nontrivial analysis in the presented text.\n- Guidance weights do not explicitly incorporate model uncertainty (despite using VAEs), and $\\gamma_t$ is fixed to 1 in experiments; ignoring variance calibration may cause mis-weighted gradients in regions with high model uncertainty.\n- Reliance on learned dynamics models introduces another layer of model bias; the framework does not quantify or adaptively mitigate this bias beyond fixed $\\lambda$, $\\beta$ schedules.\n- Missing comparisons to closely related constrained or dynamics-aware diffusion approaches that also inject dynamics into reverse sampling, such as DPCC (projection with dynamics), Equality-Constrained Diffusion (augmented Lagrangian for dynamics constraints), and Constrained Diffusers (primal-dual/ALM constrained denoising with an inverse dynamics layer). These are natural baselines for the core claim of “dynamics alignment.”\n- No analysis of dynamics model calibration or accuracy (e.g., negative log-likelihoods, calibration curves, or how guidance reacts to mis-specified models). This is critical to substantiate the claim that alignment reduces compounding error rather than simply steering toward another biased manifold.\n- The paper proposes two standalone variants (forward vs inverse) but does not evaluate their combination, which seems straightforward under the unified guidance and could be stronger.\n- Details about the return predictor training (data splits, targets, normalization, potential leakage) are not fully specified; in MuJoCo, whether reward is computed analytically or via a learned model is ambiguous.\n- The “dynamics inconsistency” metric is not fully operationalized beyond $L_2$ distances; whether this is measured against the true simulator or learned models differs across sections, and the choice materially affects conclusions.", "claims_and_evidence_yn": "No", "claims_explanation": "While the paper presents substantial empirical results, several claims lack adequate support:\n- **Unsupported theoretical claim:** The paper lists \"Theoretical Insight: We provide an analysis showing that explicit dynamics alignment mitigates compounding errors\" as a main contribution. However, no formal analysis, proofs, or non-trivial theoretical results are provided—only empirical observations (Figure 3b). This overclaims the contribution.\n- **Missing critical baselines:** The core claim of \"dynamics alignment\" is not compared against closely related constrained diffusion methods (DPCC, Equality-Constrained Diffusion, Constrained Diffusers) that also inject dynamics into reverse sampling. Without these comparisons, the claimed novelty and superiority cannot be fully evaluated.\n- **Methodological inconsistency regarding uncertainty:** The authors motivate the use of VAE-based dynamics models specifically to \"capture uncertainty\" and enable \"uncertainty-aware consistency guidance\"1. However, the experimental setup contradicts this by using a fixed uniform weighting ($\\gamma_t=1$). This effectively discards the uncertainty information the VAE provides, rendering the claim of \"probabilistic\" or \"uncertainty-aware\" guidance unsupported by the actual implementation.\n- **Unverified experimental details:** The reported 100-150 random seeds is unusually high for D4RL evaluations (typical is 3-10 seeds). This requires clarification, as it affects the credibility of the statistical claims.\n- **Missing model validation:** No calibration or accuracy metrics are reported for the learned dynamics models, yet the entire framework depends on their fidelity. This gap undermines confidence that the method reduces compounding error rather than introducing different biases.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a timely and practically relevant problem at the intersection of two active research areas: diffusion models for decision-making and offline reinforcement learning. The phenomenon of \"dynamics inconsistency\", where diffusion-generated trajectories appear globally coherent but violate local transition constraints, is a recognized failure mode that practitioners encounter. The proposed solution of integrating forward/inverse dynamics guidance into the denoising process is conceptually simple, easy to implement atop existing diffusion planners, and broadly applicable. Researchers working on diffusion-based control, model-based RL, robotics planning, and offline RL would likely find the framework and empirical findings relevant to their work.", "requested_changes": "Refer to the weaknesses above.", "broader_impact_concerns": "Since the paper motivates the work with safety-critical applications like medical treatment and autonomous driving, a Broader Impact Statement should explicitly address the risks of relying on learned dynamics models for safety constraints, where model errors could lead to dangerous failures. Additionally, the significant computational cost of diffusion-based planning  warrants a discussion on energy consumption and the environmental impact of large-scale deployment.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)