Consistency Trajectory Planning: High-Quality and Efficient Trajectory Optimization for Offline Model-Based Reinforcement Learning
Authors: Guanquan Wang
Takuya Hiraoka
Yoshimasa Tsuruoka
Venue: Accepted by TMLR ()

Abstract: This paper introduces Consistency Trajectory Planning (CTP), a novel offline model-based reinforcement learning method that leverages the recently proposed Consistency Trajectory Model (CTM) for efficient trajectory optimization. While prior work applying diffusion models to planning has demonstrated strong performance, it often suffers from high computational costs due to iterative sampling procedures. CTP supports few-step trajectory generation without significant degradation in policy quality. We evaluate CTP on the D4RL benchmark and show that it consistently outperforms existing diffusion-based planning methods in long-horizon, goal-conditioned tasks. Notably, CTP achieves higher normalized returns while using fewer denoising steps. In particular, CTP attains comparable—or even superior—performance with reduced inference cost, highlighting its practicality and effectiveness for high-performance, low-latency offline planning.

Published in Transactions on Machine Learning Research (12/2025)
Consistency Trajectory Planning: High-Quality and Efficient Trajectory Optimization for Offline Model-Based Reinforce- ment Learning
Guanquan Wang
Department of Information and Communication Engineering The University of Tokyo
Takuya Hiraoka
takuga-h @nec. com
NEC Corporation, Tokyo, Japan
Yoshimasa Tsuruoka
Department Information and Communication Engineering The University of Tokyo
Reviewed on OpenReview: https: // optp//veeee........mm/ net/ /ol/ffooo forum? id= RVCKT9ISVJ
Abstract
This paper introduces Consistency Trajectory Planning (CTP), novel offline model-based reinforcement learning method that leverages the recently proposed Consistency Trajec- tory Model (CTM) for efficient trajectory optimization. While prior work applying diffu- sion models to planning has demonstrated strong performance, often suffers from high computational costs due to ilerative sampling procedures. CTP supports few-step trajec- tory generation without significant degradation in policy quality. We evaluate CTP on the D4RL benchmark and show that consistently outperforms existing diffusion-based planning methods in long-horizon goal-conditioned tasks. Notably, CTP achieves higher normalized returns while using fewer denoising steps. In particular, CTP attains compara- ble or even superior performance with reduced inference cost, highlighting its practicality and effectiveness for high-performance, low-latency offline planning.
1 Introduction
Recent advances in generative models have significantly impacted offline reinforcement learning (RL), espe- cially in trajectory planning where agents must learn optimal behavior from fixed datasets. Diffusion-basel methods such as Diffuser (Janner et al. 2022 and Decision Diffuser (Ajay et al. 2023) have proven effec- tive for modeling complex trajectory distributions, but their reliance on iterative denoising makes real-time decision- makinn impractical.
To address the inefficiencies inherent in iterative diffusion-based sampling, recent work has explored distilled models such Consistency Model (CM) (Song al.] 2023), which bypasses the reverse diffusion process by directly learning the mapping from noise to data (Wang al. 2023 Kang et al. 2024). In the context of model-free RL, CM has demonstrated substantial speedup with only marginal performance degradation (Ding & Jin 2024 Wang et al., 2024a).. However, a critical limitation remains: CM lacks a principled mechanism to trade ofl between sampling speed and sample quality. This stems from the nature of the distillation process, where the consistency function trained to project arbitrary intermediate states along the ODE trajectory back the clean data. As a result, practice, the multistop sampling procodure of CM for improved sample quality alternates between denoising and injecting noise. This iterative refinement, however, accumulates errors particularly the number of function evaluations increases.

---

Published in Transactions on Machine Learning Research (12/2025)
Building upon this observation, Consistency Trajectory Model (CTM) has recently been proposed generalization of both score-based and consisteney-based models (Kim et al. 2024). CTM enables anytime- to-anytime transitions along the probability flow ODE, supporting flexible and efficient generation through both short-step and long-jump sampling. CTM retains access to the score function while allowing diverse training losses such as denoising score matching and adversarial losses, ultimately improving expressiveness and generalization.
recent concurrent study (Duan et al. 2025) also investigates the use of CTM for improving inference efficiency in offline RL. The algorithm operates in a model-free setting and primarily focuses on policy learning from demonstration data. Although this algorithm accelerates diffusion-based models for decision-making tasks, it offers limited improvements in long-horizon tasks.
Motivated by the strengths of CTM and the need for efficient planning in offline RL, we propose Con- sistency Trajectory Planning (CTP), a novel offiine model-bases RL algorithm that integrates CTM into the trajectory optimization process (Section CTP inherits the speed and flexibility of CTM, allowing planners to efficiently navigate the trado-off between planning speed and return quality. Unlike previous score-based planners requiring classifier guidance or iterative refinements, our method enables near single- step sampling (one a few denoising steps depending on task complexity) while retaining controllability and sample diversity. Furthermore, CTM's access to score information enables conditional planning and return-conen neturn-condiioonng without the need for learned Q-functions, which faces challenges due to overestimated Q-values for out-of-listribution actions (Kumar et al. 2020 Levine et al. 2020
We evaluate CTP on D4RL benchmark tasks (Fu et al. 2020) (Section 5 Across multiple tasks, CTP
consistently matches or outperforms prior diffusion- bassed planners and consistency policies while achieving significant improvements in inference speed-mald it well-suited for real-time high-frequency control applications.
2 Related Work
Diffusion models. Diffusion models generate data by reversing a progressive noising process Sohl-Dickstee
et al. 2015: Ho et al. 2020). This view unifies with score-based modeling and admits SDE/ODE formulations
(Hyvärinen & Dayan 2005 Song et al., (021)), also connecting to mmery--based learning via scores as log-
density gradients (Du & Mordatch 2019 Nijkamp et al.) 2019 Grathwohl et al. 2020 Conditioning
is commonly implemented via guidance: classifier guidance (Nichol & Dhariwal 2021 and classifier-free
guidance (Ho & Salimans 2021). These techniques underlie stale-of-the-art image/ text generation (Saharia
et al. 2022 Nichol & Dhariwal
2021).
Diffusion Models for Policy Representation. Diffusion models have been adopted to represent expres- sive policies in RL (Pearce el al. 2023 Chen et al., 2023). Diffusion-QL, (Wang el al.] 2023) incorporates diffusion models into both Q-learning and Behavior Cloning capture multi-modal action distributions, with later work improving efficiency via action approximation to avoid repeated denoising during training (Kang) et al. 2024h. Other work includes robotic visuomotor diffusion policies with preeding-horzzon control and visual conditioning (Chi et al. 2023); g; IDQL., which reinterprets TQT, as actor-critic and extracts policies from diffusion-paraan pareeetriie behavior models Mlansen-Esstruce et al. 2023); QSM, which links diffusion-policy scores to Q-gradients for principled updates (Psenka et al.) 2024]: DQS, which samples from Boltzmann policies via diffusion (Jain et al., 2 and DACER, which applies diffusion policies to online actor-critic with GMM-basod entropy estimation (Wang et al. 2024b).
Diffusion Models for Trajectory Optimization. In addition to representing policies, diffusion models have also been employed as trajectory generators for planning in offline RL (Liang et al., 2023 Du et al. 2023 Yang et al. 2023 Li ct al. 2023 Chen et al. 2024 Dong et al. 2024). A notable example is Diffuser (Janerr el al. 2022), which learns to generate entire trajectories from oflline data and applies guidance lo bias the trajectories toward high returns or task-specific goals. At execution time, however, only the first action of each generated trajectory applicd, after which the model replans in a recoding-horizon fashion. Decision Diffuser (Ajay et al.) (023) follows a similar idea but conditions trajectory generation directly on farget returns or goals, eliminating the need for an auxiliary reward model.
2

---

Published in Transactions on Machine Learning Research (12/2025)
Summary. Across behavior cloning, actor critic, energy-based policies. and online control CChi et al,
2023 Hansen-Estruch et al., 2023 Psenka et al. 2024 Jain et al. 2025 Wang et al. 2024b), diffusion- based methods typically require multi-step denoising or gradient refinements, incurring nontrivial inference latency; trajectory planners like Diffuser/Decssion Deccison Diffuser further amplify cost by generating full sequences and replanning. While Diffusion-QI. variants reduce some training-lime overhead, they do not. fundamentally climinate the need for iterative reverse diffusion processes during the inference process.
Positioning of our work. Beyond diffusion- based planners, the work most closely related to ours is Consis- tency Planning (CP) (Wang et al. 2024a), which first introduced the use of consistency models for trajectory optimization in offline RL. Both CP and our proposed CTP share the same high-level goal of trajectory-level planning; however, CTP introduces several key differences. First, instead of using standard CM that only learns mappings from It to Io, CTP leverages CTM which support mnytime-to-anytimg mappings along the probability flow ODE. This formulation enables greater flexibility at inference time and reduces reliance on iterative guidance. Second, whereas CP steers trajectory generation using classificr-free guidance, CTP adopis critic-based selection mechanism that evaluates multiple candidate plans and selects the best according to predicted returns (Chi et al. 2023). This avoids the potential instability of classifier based guidance while aligning the planner directly with the RL objective. In addition, CTP incorporates several architectural refinements, including basede-based trajectory representation and transformer backbones, which further improve scalability to complex benchmarks. Overall, these modifications allow CTP to handle more challenging tasks while maintaining the efficiency advantages of the consistency framework.
Motivated by these limitations, our work introduces more efficient alternative based on the recently pro- posed CTM, which supports Maze2D, and two steps for harder benchmarks). By integrating CTM into the trajectory optimization process, our method circumvents the need for iterative denoising and enables fast planning with minimal performance degradation. In contrast to prior diffusion-based planners, CTP achieves a favorable trade-off between sample quality and computational cost, making it particularly well-suited for timo-sensitive offline RL applications.
few denoising steps, depending on task complexity (single-step
3 Preliminary
3.1 Reinforcement Learning Problem Setting
We consider standard reinforcement learning formulated as Markov Decision Process (MDP) M (S, A, P. R, 7,, do), where S is the state space, A is the action space, P is the transition function, R is the reward, E 1) is the discount factor, and do is the initial state distribution. The objective is to learn a policy that maximizes the expected discounted return E Dlmdd* (Sk, ak)) where kend is the index of final time step in trajectory.
3.2 Consistency Trajectory Models
Diffusion models generate samples by gradually adding and then removing Gaussian noise, but require hundreds of denoising steps at inference, limiting their practicality in RL. CMs (Song et al. 2021 Kim et al. 2024h address this by learning a direct mapping from noisy input x, at arbitrary noise level \ back to its clean counterpart X,, enabling generation in one or a few steps. This efficiency makes CMs appealing for planning tasks where inference speed is critical.
CTMs extend the idea of CMs (Song et al.] 2021 Kim et al. 2024 from data generation to trajectory prediction. The framework follows teacher student distillation paradigm. The teacher model D& is trained as denoiser, mapping noisy data back to clean ones:
stuchet-student
C(() :- -Borppann. To n
(1)
where the noise level is sampled from log-normal distribution Perain following Karras al.
2022
3

---

Published in Transactions on Machine Learning Research (12/2025)
To stabilize training, D adopts skip-connection parametterization:
Do(xe,t) Cakip (t) X + Cout (f) (x,,),))
(2)
where F& is neural network and Cakip Cout time-dependent coefficients that ensure correct boundary behavior at t = These coefficients are chosen such that Cakip (e) = 1 and Cout(e) = 0, ensuring that the model exactly reproduces the clean data at the boundary = E, i.e., Do Xo-
The student model Go (Eq. 3 learns to map noisy inputs at arbitrary t to less noisy versions at w < t, combining an identity shortcut with neural residual.
Ge (x,,t,w) x + (1 (1-7) g0 (xx, w),,
w)
(3)
Distillation enforces local and global consistency by comparing student predictions with targets obtained from the teacher through short ODE integrations, typically solved with a Heun method. This yields the CTM loss (Eq. 4 while denoising score matching (Eq. regularizes cases where w Re t.
LCTM(O: Φ) == Bu Exco Exe lxo [d (Xtarget (xe,t, u, w) Xest (x,,t,w))
(4)
where d(:--) is the squared l distance with d(r,y) 11(x - v) ls and Xest (xe, 4, w) Gog(o) (Go (X,,t, w) 0), Xtarget (x, t, (.u, w) Gag(o) (Grarget w, 0), representing the predictions at time 0.
LDMM (0) = Exxxo Exelx ||xx0 go (x1, L 1112]
(5)
which acts as a regularization mechanism that strengthens the learning signal and improve the accuracy when w
To further improve sample quality, adversarial discriminator added, yielding GAN loss (Eq. 6
LGAN(O, 1)) = Exx [log dry (xo)] + Bwejo. Exs |xa (log (1 - dny (Xest (X1, w)))],
(6)
where dn is the discriminator function, and n denotes its parameters.
The overall training objective combines these components (Eq. and the complete training algorithm is summarized in Appendix
((0,n) == (CTM(A;:) + ADSM + ACAN
4 Planning with Consistency Trajectory Model
This paper explores the integration of CTM into the planning architecture in offline RL. In the following, we discuss how we use CTM for the trajectory optimization process. Section 4.1 details the training process of each component, and Section 4.2 describes how the consistency trajectory planner is applied during inference.

---

Published in Transactions on Machine Learning Research (12/2025)
4.1 Training process
Trajectory representation. As outlined by Ajay et al. 2023 the diffusion process encompassss only the state transitions, as described by
X1. (T) = (Sk: M (R+
(8)
In this notation, & indicates the timestep of a state within a trajectory т, H represents the planning horizon, and t4 E tv] is the timestep in the diffusion sequence. To make the model look ahead farther, we choose a jump-step planning strategy (Lu et al. 2025). Jump-step planning models H x M environment steps, where M N+ is the planning stride. Consequently, xx, (T) is defined as a noisy sequence of states, represented as a two-dimensional array where each column corresponds to a different timestep of the trajectory. In the training process, the sub-sequence t; follows the Karras boundary schedule (Karras al. 2022):
1/8 + i-1
=
(9)
where 0.002, tN 80, and p=
Inverse dynamics model training. To derive actions from the states generated by the diffusion model, employ inverse dynamics model (Agrawal et al. 2016 Pathak et al. 2018), denoted as hep trained using ($k, MK+M) as input, dg as target, sampled from the dataset D consisting of trajectories. Therefore, actions can be obtained via the inverse dynamics model by extracting the state tuple ($k, 8 at diffusion timestep to:
C(()) (L) |lax lapss, 8R+
(10)
CTM training. Following the general framework in Section 3.2 we instantiate CTM in RL using (T) as noisy sequences (Eq. Sb. The teacher D& is trained to map these noisy trajectories back to clean ones using the objective in Eq. 1 and knowledge distilled to the student Go via the CTM and auxiliary losses (Eqs. This adaptation allows CTM to directly generale candidate state sequences conditioned on the current state 8k, which are later ranked by the critic model.
Critic Model training. During the sampling process, use Monte Carlo sampling from selections, where C selections are firstly sampled from CTM candidates. To select the best plan from the C selections, critic model Va is trained using (T) input, the accumulated discounted returns R3 as target output, where a denotes the parameters of the critic network. Specifically, R& is calculated by
R
> Tk+hh-
(11)
4=0
To train the critic model Va, we minimize the mean squared error between the predicted return and the actual accumulated return:
Cerito(() = ETnD (VV((x((F)) - 10x)+x)) Rn)
(12)
This value-based selection process enables the planner to rank candidate trajectories according to their expected returns and choose the highest-scoring plan for execution.
4.2 Inference process
Sampling with CTM. To generate samples using CTM, we follow a reverso-time denoising procodure along predefined sequence of time steps to. .t....,ty where tn ty and to = c. Sampling begins by drawing an initial noisy sample Xty from a standard Gaussian prior NO T). Then, for each time step
5

---

Published in Transactions on Machine Learning Research (12/2025)
8k++M M
M)M
consistency trajectory model
times
best plan
8k+-M
SK+(H 1JM
8k+M
8k+ 2M
(H 1)M
critic model Va
inverse dynamies
expected cumulative rewards
Figure Consisteney Trajectory Planning. Given the current state 8k, Consistency Trajectory Planning generates O sequences of future states with planning horizon H. Then, the best plan is selected by the critic model. Finally, the inverse dynamics model used extract and execute the action og from sk and 8&+-M which are from the selected best plan.
- 0, the model applies single-step denoising operation using the student model Ge, which maps the current noisy sample Xt,+1 from time tnt1 to less noisy representation at time tx
Formally, this process iteratively computes
Xt - Go(Xtnn1 Inti, bn),
(13)
until reaching the final output Xt. which serves as the generated sample. Unlike the inference process CM, we do not add noise perturbations intermediate samples, since such perturbations would cause error accumulation. This deterministic sampling procedure allows CTM to produce high-quality samples in a small number of steps, while proserving flexibility in the choice of discretization schedule.
Algorithm Consistency Trajectory Planning
1: Input: Planning horizon H, Dataset D, Discount factor ?, Candidate number C, Planning stride M
2: Initialize: Diffusion Transformer Planner D&, Consistency Trajectory Planner ge. Diffusion Inverse dynamics hip Critic Va
3: Calculate accumulated discounted returns R4 = Endd T* for every step k
4: function TRAINING
Sample 8k, 8k+M (H- (k; 0k+ M,+ (H+(H-1)MM Rs from
Train diffusion model D& using 8& as condition and S...t.....AHHH-.)) as target output (Eq.
Distill consistency trajectory planner go (Eq. 7
Train inverse dynamics hap using $k, SM+M input, dş target output (Eq. 10
Train critic Va using 8
input, R as target output (Eq. 12
10: end function
11: function PLANNING(s)
12: Randomly generate C' plans using CTP sampling, while fixing the first state as during sampling
13: Select the best plan using critic Va
14: Use the inverse dynamics he to generate action using and the next state in the best plan
15: end function
СТР Inference. During the inference process, we first observe a state s in the environment and sample a Gaussian noise X Then, CTM iteratively predicts the denoised trajectorics x,, from the noisy inputs. This process is repeated C times to generate C plans using ge, while the first state s is fixed during sampling. Then, select. the best plan using critic V.. Finally, extract states (sk, 8K+M) from the denoised

---

Published in Transactions on Machine Learning Research (12/2025)
trajectory and get the action ag via our inverse dynamics model hg. Algorithm 1 and visualized in Figure
The algorithm of CTP is provided in
5
Experiment
In this section, we present the experiment environment, experiment setting and report empirical results that validate the effectiveness of the proposed CTP algorithm across diverse offline RL tasks.
5.1 Experiment Environment
We train the diffusion model, inverse dynamics model, and consistency trajectory model on publicly available D4RT datasets. Evaluation is conducted across diverse Gym tasks, including locomotion (HallCheetah, Hopper, Walker2d), long-horizon planning (Maze2D), and high-dimensional robotic control tasks (Antmaze, Kitchen, Adroit) from the D4RL benchmark suite (Fu et al., (2200). These lasks are characterized by continuous state and action spaces and are conducted under offine RL settings. More details of the tasks are provided in Appendix A
Locomotion The locomotion lasks Hopper, HallCheetah, and Walker2d are widely adopted due to their controlled dynamics, reproducibility, and varying levels of complexity. These environments are based on the MuJoCo physics engine (Todoroy et al., 2012) and simulate planar bipedal or quadrupedal agents that must learn to move forward efficiently.
Maze2D To validate the long-horizon planning capabilities of CTP, we conduct an evaluation in the Maze2D enviromment (Fu et al. 2020), where the task involves navigating specific goal location, with reward of 1 assigned only upon reaching the goal. Because it requires hundreds of steps to reach the goal, even the most advanced model-free algorithms struggle with effective credit assignment and consistently reaching the goal.
AntMaze The AntMaze task extends the Maze2D environment by replacing the simple 2D ball agent with more complex quadrupedal "Ant" robot, thereby combining challenges of both locomotion and high-level planning. In the diverse variant of the dataset, the ant is initialized at random positions and directed toward randomly sampled goals. In contrast, the play variant consists of trajectories guided toward manually selected goal locations within the maze.
Kitchen The Kitchen environment simulates a robotic manipulator interacting with various appliances in a realistic kitchen setting, requiring the agent to perform multi-slage manipulation tasks to accomplish specified goals. In the partial dataset, only a subset of trajectories successfully achieves the full task, allowing imitation-based methods benefit from selectively identifying informative demonstrations. The mixed dataset, the other hand, contains no trajectories that solve the lask in its entirely, necessitating the of RI. to stitch together relevant sub-trajectories.
The Adroit Hand benchmark consists of four ligh-degroo-offrroodo manipulation tasks constructed from motion-capture demonstrations of human hand movements (Rajeswaran et al. 2018 Fu et al. 2020). The benchmark couples challenging planning objectives with low-level motor control, thereby providing a stringent test of our approach.
Adroit
5.2
Experiment Setting
We evaluate the performance of our proposed method by comparing it against both actor-critic-based ap- proaches, including Consistency Actor-Critic (C-AC) (Ding & Jin 2024) and Diffusion-QL (D-QL) (Wang
et al. 2023). as well as model-basee based planning methods, such as Diffuser JJanerr et al., 2022), Decision Dif-
fuser (DD) (Ajay et al. 2023), Consistency Planning (CP) (Wang et al. 2024ab, Reward-Awai Consistency
Trajectory Distillation (RACTD) (Duan et al. 2025 and Lower Expectile Q-learning (LEQ) Park & Lee
2025)_

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: The average scores of Diffuser, Decision Diffuser, Diftusion-QL, Consistency-AC. Consistency Plan- ning and our method on D4RL locomotion tasks are shown. The results of previous work are quoted from Ding & Jin (2024). Ajay al. (2023) and Wang et al. (024a)..
Dataset
Environment
Diffuser
DD
D-QL
C-AC
CP
СTP
Medium Eepiumt-xppert
Halfcheetah
79.8
90.6
96.8
84.3
94
89.3
Medium-Replay
Halfcheetah
42.2
39.3
47.8
58.7
40.6
4..4
Medium
Halfcheetah
44.2
49.1
51.1
69.1
46.8
5.14
Medium-Exee Exppett
Hopper
107.2
111.8
111.1
100.4
107.5
1.2
Medium-Replay
Hopper
96.8
100
101.3
99.7
97.8
90.0
Medium
Hopper
58.5
79.3
90.5
80.7
87.8
83.
Medium- Expert
Walker2d
108.4
108.8
110.1
110.4
109.8
110.1
Medium Seeiium-Replyy
Walker2d
61.2
75
95.5
79.5
75.3
86.9 0.3
Medium
Walker2d
79.7
82.5
87.0
83.1
80.5
85.7 0.2
Average
75.3
81.8
87.9
85.1
82.2
83
Table 2: The performance of CTP, Diffuser, and previous model-free algorithms in the Maze2D environment,
which tests long-horizon planning due to its sparse reward structure. The results of previous work quoted
from the data provided in Janner et al. (2022). Wang et al. (2024a) Lu et al. (0225) and Duan et al. (0025)
Dataset
MPPI
CQL
IQL
Diffuser
CP
RACTD
D-QL
СТР
Maze2D U-Maze
33.2
3.7
47.4
113.9
122.7
125.7
140.6
154.1 2.3
Maze2D Medium
10.2
5.0
34.9
121.5
121.4
130.8
152.0
167.1 + 2.4
Maze2D Large
5.1
12.5
58.6
123.0
119.5
143.8
186.8
216.7 + 3.4
Average
16.2
7.7
47.0
119.5
121.2
133.4
159.8
179.3
For all experiments, we report results as the mean over 150 independent planning seeds to ensure statistical robustness. Following the evaluation protocol established in Fu et al. 2020 we adopt the normalized average return as the primary performance metric.
Unless otherwise specified, the CTP algorithm employs N denoising steps, which found to yield near-saturated performance across most tasks. An exception is the Maze2D domain and Walker2d, where we use single denoising step (N 1) due to the relative simplicity of the environment. We use N = 3 steps for the Kitchen domain. For comparison, competing methods employ different denoising schedules: the diffusion policy utilizes N = 5 steps (Wang et al. 2023). Diffuser adopts N = 20 steps, and Decision Diffuser uses N = 40 steps (Janner et al. 2022 Ajay et al., 2023). Consistency- based baselines such as Consistency Actor-Critic and Consistency Planning are evaluated with N denoising steps, consistent with their original implementations. More implementation details provided in Appendix
5.3 Experimental Results
As shown in Table | CTP achieves competitive performance across standard locomotion benchmarks. These results highlight the effectiveness of CTP in matching the performance of prior state-of-the-ard diffusion-based planning algorithms.
Given the relatively lower complexity of Maze2D comparod to AntMaze and Kitchen, our model capable of generating effective trajectories with onc-step generation (N=1), while still achieving best performance
(Table 2 Notably, when compared to the recent concurrent. method RACTD (Duan et al. 2025 and D-
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: The performance CTP, Diffuser, and previous model-free algorithms in the Kitchen environment, which tests both locomotion and high-level planning capability. The results of previous work are derived from the data provided in Ding & Jin 2024) and Ajay al. (0023
Dataset
Environment
Diffuser
DD
D-QL
C-AC
СТР
Mixed
Kitchen Kitchen
47.5
65 + 2.8
62.6 5.1
45.8 1.5
14.5000 91.2 + 1.0
Partial
33.8
57 + 2.5
60.5 + 6.9
38.2 1.8
Average
40.65
61.55
82.85
Table 4: The performance of CTP, D-QL and LEQ in the Antmaze environment. The results of D-QL and LEQ are quoted from (Wang et al. 2023 and Park & Lee (2025)
Dataset
Environment
LEQ
D-QL
СТР
Diverse
Antmaze-Large
60.2 - 18.3
56.6 7.6
82.0 = 3.1
Play
Antmazo-Targe
62.0 9.9
46.4 8.3
82.0 3.1
Diverse
Natmaze-Medium
46.2 + 23.2
78.6 10.3
86.0 2.8
Play
Antmaze-Medium
76.3 17.2
76.6 10.8
83.3 3.0
Average
61.18
64.6
83.33
QL (Wang et al. 2023 which adopt model-free approach, CTP consistently achieves higher returns across all Maze2D tasks, which require accurate trajectory optimization and modeling long-term dependencies. By leveraging environment dynamics through CTM, CTP enables high-quality planning under sparse-reward, long-horizon scenarios.
3 and Z prosent the performance of CTP on the Kitchen and AntMaze benchmarks, illustrating its robustness in addressing complex, goal-conditioned control lasks. Our experimental results demonstrate that CTP performs competitively not only in relatively simple environments such as Maze2D-wii no robotic actuation is involvod-bil also in more challenging domains like Kitchen and AntMaze, which demand high-dimennsionll, temporally coordinated control.
Tables
We further evaluate CTP on the "expert" dataset from the Adroit benchmark, which consists of 5000 trajectories generated by policy that consistently completes the task. Quantitative results reported in Table 5 While CTP does not outperform all prior methods across every task, it achieves competitive results overall. These findings suggest that CTP is effective in goal-conditioned manipulation scenarios that require precise spatial reasoning and long-horizn planning.
While CTP achieves competitive overall results, we note that model-free Diffusion-QL obtains stronger scores on several MuJoCo locomotion tasks (Table [ This performance gap can be explained by the nature of these benchmarks. First, locomotion tasks such as HalfCheetah, Hopper, and Walker2d primarily reward short-horizon control signals (e.g., maintaining balance and maximizing forward velocity), which can be effectively optimized through direct policy gradient updates and critic-based kemporal-different learning (Wang et al. 2023 Ding & Jin 2024). In contrast, model-based planning approaches like CTP introduce additional sources of approximation error due to dynamics modeling and trajectory sampling, which may be unnecessary overhead in these relatively simple tasks. Second, trajectory-Ievel planning with small candidate counts may under-explore the fino-grained action variations that are critical for maximizing continuous control rewards in MuJoCo environments. Third, the critic in CTP is trained to evaluate full trajectories rather than single-step actions, which can be less sample-efficient for dense-reward settings compared to

---

Published in Transactions on Machine Learning Research (12/2025)
Table 5: The performance CTP against other baselines in the Adroit environment. The results of previous work are quoted from the data provided in He al. (2024) and Lu et al. (2025).
Dataset
Environment
BC
BCQ
CQL
IQL
AlignIQL
D-QL
CTP
Expert
door
34.9
99.0
101.5
103.8
104.6
104.3
104.1 044.1:0.3
Expert
hammer
125.6
107.2
86.7
116.3
124.7
55.9
110.5 3.1
Expert
pen
85.1
114.9
107.0
111.7
116.0
60.9
104.1 = 5.3
Expert
relocate
101.3
41.6
95.0
102.7
106.0
108.8
108.8 + 0.6
Average
86.7
90.7
97.6
108.6
112.8
82.48
106.88
Average walker2d-me
Average
walker2d-m
Average
walker22dmr
120
120
orffuser (Avg, Score)
Difuserr (Agg Score)
Diffuser (Avg. Scorel
CTP (Aug. Score)
CTP (Avg. Score)
CTP (Avg. Score)
100
Saturates [=1]
Avg. Scare
80
Saturates with smalleri (-11
Avg. Score
80
Saturates with smaller N (-1)
40
40
104
10a
(Log Scale)
N (Log Scale)
(Log Scale)
(a)
(b)
(c)
Inference Time vs. N on w walker2d me
Inference Time (ms per sample)
Inference Time vs. N on walker2dnn
Inference Time (ms per sarrile)
Inference Time vs. N on walker2d mr
2000
(inference
2000
Diffuser (Inference
2000
Orference
CTP (interence Time!
CTP Unterence Time)
CTP (Inference Time!
1500
1500
1550
1000
1000
1000
500
Larger time
500
Exp 100
500
Larger time gap
10a
10
10"
10
10'
10'
(Log Scale)
(Log Scale)
(Log Scale)
(d)
(e)
(f)
Figure 2: Comparison of the average normalized scores (top row) and inference times (bottom row) of СТР and Diffuser on three walker2d datasets: waller22t-mediun-exper walker2d-mediun,, and walker2d- medium-replay, Each column corresponds to a specific dataset. The x-axis denotes the number of denoising steps N on a logarithmic scale. Vertical error bars represent the standard deviation over five random seeds. While CTP achieves near-optimal performance with significantly fewer denoising steps (e.g., saturating N 1), Diffuser requires substantially more steps to match similar scores. Inference time grows rapidly with larger N for Diffuser, whereas CTP maintains consistently low inference time.
actor-critic methods tailored for stepwise feedback. Together, these factors suggest that the gap arises not from a fundamental weakness of CTP, but from mismatch between algorithmic design and the structural simplicity of MuJoCo lasks.
Computational Time. denoising step numbers N, we conduct a systematic evaluation of both inference time and policy performance on the walkerrdd-medium--xppr task. Since both CTP and Diffuser are built upon generative modeling frameworks based on probability flow, their inference cost scales with the number of denoising steps N. However, by design, the CTP enables effective few-step sampling, in contrast diffusion models which typically require iterative refinement to achieve comparable sample quality. Figure 2 presents detailed comparison of the average normalized scores and inference time (in milliseconds per sample) across varying
To assess the computational efficiency of CTP relative to Diffuser under varying
10

---

Published in Transactions on Machine Learning Research (12/2025)
denoising steps. Each data point reports the mean and standard deviation computed over five random seeds. As shown, CTP achieves near-optinal performance with N = 1, and saturates fully by N = 2. In contrast, Diffuser requires up to N = 20 steps to reach its performance plateau, with inference time increasing siguificantly with each additional step. Furthermore, Diffuser with N = 20 requires approximately 1900 mss per sample for inference, while CTP with N 1 achieves the same result. within approximately 15 ms, demonstrating its substantial efficiency under this setting. Despite this large discrepancy computation time, CTP continues to outperform Diffuser in terms of poliey quality, yielding slight improvement in also evaluate inference latency large-scale tasks
normalized score. To further validate scalability, beyond locomotion. Figure X reports results on AntMaza-medium-ddvee
and Kitchen-mixed tasks. It is evident that CTP consistently outperforms Diffuser across both AntMaze and Kitchen domains for the same number of denoising steps N. This implies that, to achieve a comparable level of performance, CTP can deliver a substantial inference speedup relative to Diffuser, underscoring the scalability advantage of CTP in both navigation and manipulation tasks.
Average Score vs. N on antmaze-md
Average Score vs. N on kitchen-mixed
125
Diffuser (vv Score)
125
Diffuser (Avg. Score)
(Avg. Score)
(Aug. Score)
Avg. Score
100
Avg. Score
100
25
25
0-
10"
N (Log Scale)
N (Log Scale)
(a)
(b)
Inference Time (ms per sample]
Inference Time (ms per samplel
Inference Time antmaze-md
Inference Time N kitchen-mixed
1000
1000
Diffuser (inference Time)
Diffuser (Inference Time)
800
Linforence Time)
[Inference Time)
600
600
400
400
200
200
10a
10a
100
10a
(Log Scale)
(Log Scale)
(c)
(d)
Figure 3: Comparison of average normalized scores (top) and inference times (bottom) of CTP and Diffuser on AntMaze and Kitchen tasks.
These results underscore the efficiency advantage of CTP, demonstrating that it achieves strong perfor- mance with substantially fewer denoising steps, thereby offering more practical solution for time-sensitive deployment scenarios.
Error accumulation. Score-based models are prone to discret ization errors introduced by SDE/ODE solvers, while distillation-bas models often suffer from error accumulation across multiple sampling steps. CTM mitigates these issucs and reduce the error accumulation from O ( + +.) to O (vtN) [Kim et al., 2024). As shown in Figure 4 the proposed algorithm CTP achieves significantly improved performance over offline RL methods based CM when the number of sampling steps N > 2.
5.4 Ablation Studies
To analyze the contributions of various design components CTP and isolate the effect of using CTMs, we conduct several ablation and controlled experiments. To isolate the benefits of consistency trajectory modeling, we conduct controlled comparisons against two existing methods: DD (which uses diffusion
11

---

Published in Transactions on Machine Learning Research (12/2025)
Average Performance vs. samplings steps
Nomalized Scare
%
70
50
Consistency Trajectory Models
Consistency Models
Sampling Steps N
Figure 4: The average performance versus sampling steps for CTP and CM on kitchen-partial dataset.
model) and CP (which uses a CM). For fairness, we adapt both methods to use the same architecture and training setup our CTP, including the Transformer- backbone, stride-based trajectory representation, value-based filtering, and training setup. We refer to these aifffeddarcchtectueee variants DD-improved and CP-improved respectively. This setup ensures that all methods differ only in the generative models used, allowing for a clean and controlled comparison. The results, shown in Table 6. demonstrate that in identical architectural and algorithmic settings, CTP consistently outperforms the DD-improved and CP-improved in terms average return in Maze2D tasks under comparable denoising steps. Corresponding results on the Kitchen domain, as well as additional ablation studies on the M-stride trajectory representation and value-based trajectory selection, provided in Appendix B
Table 6: Average normalized scores and inference time across different denoising steps N for CTP, CP- improved, and DD-improved in Maze2D environments.
Method
N
Average Score
Inference time (ms)
Maze2DD-arge
MazeiD-MMedium
Maze2D-Umaaee
218.1 + 3.5
167.1 2.4
154.1 + 2.3
9.6 4 0.1
СТР
2
212.9 + 2.8
165.5 2.7
149.9 + 2.5
11.7 0.1
3
220 + 3.5
169.9 + 2.4
156.2 + 2.2
13.6 + 0.1
185.1 +: 4.5
150.8 3.3
147.8 +: 3.0
10.0 1.00:... 0.2
CP-improved
2
204.1 + 4.0
153.8 + 2.8
145.5 + 3.1
11.5 0.1
61.2 + 7.3
91.8 + 7.4
20.5 + 6.3
13.2
182.7 + 5.4
106.6 5.4
113.9 + 5.1
9.5 0.2
DD-improved
203.5 + 3.5
160.7 2.7
146.1 3.1
23.0 0.2
10
207.5 + 4.1
159.0 2.8
142.7 3.0
40.9 + 0.2
6 Conclusion
In this work, we propose CTP, novel approach that integrates CTM into the trajectory optimization framework for offline RL. CTP enables highly effective trajectory sampling with only one or a few denois- ing steps, achieving superior performance across standard benchmarks. Our method demonstrates robust generalization in both simple and complex environments, showcasing its practical value for efficient policy planning.
Looking ahead, several avenues of improvement remain. First, enhancing the critic model (Kostrikov et al, 2022 to better estimate the upper bound of plausible returns cettunnsraaterr rather than the mean improve planning guidance; this can be achieved using asymmetric loss functions such as the expectile loss. Second, the consistency loss weighting scheme can be refined to more accurately reflect the impact of different noise scales during training [Song & Dhariwal 2024), thereby amplifying the signal from informative samples. Finally, incorporating advanced network architectures (c.g., UNet, Transformer) along with curriculum lcarning strategies may further improve training stability and policy performance. We leave these directions for future work.
12

---

Published in Transactions on Machine Learning Research (12/2025)
References
Pulkit Agrawal, Ashvin V Nair, Picter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in Neural Information Processing Systems, 29, 2016.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit, Agrawal. Is condi- tional generative modeling all you need for decision makinn? The Eleventh International Conference on Learning Representations, 2023.
Chang Chen, Fei Deng, Kenji Kawaguehi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. In The Twelfth International Conference Learning Representations. The International Conference Learning Representations (ICLR), 2024.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high- fidelity generalive behavior modeling, 2023. URL
http///awv..orr/a////0////// 14548
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilum Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 027836442411233668, 2023.
Zihan Ding and Chi Jin. Consistency models rich and efficient policy class for reinforcement learning. The Twelfth International Conference on Learning Represenlations, 2024.
Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lv, Changjie Fan, and Zhipeng Hu. Aligndiff: Aligning diverse human preferences via behavior-customisable diffusion model. ICLR, 2024.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Newral Information Processing Systems, 32, 2019.
Yikun Du, Sherry Yang, Bo Dai, Hanjum Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuumans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in neural information processing systems, 36:9156 9172, 2023.
Xintong Duan, Yutong He, Fahim Tajwar, Ruslan Salakhutdinov, J Zico Kolter, and Jeff Schneider. Ac- celerating Diffusion Models Offline RI. via Reward-Aware Consistency Trajectory Distillation. arXiv preprint XrXi:22006.0 07822, 2025.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. DARL: Datasets for Deep Data- Driven Reinforcement Learning. CoRR, abs/2004.07219, 2020. URL https://arxiv.org// wrgg/bs/////.......
07219
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In Proceedings of the International Conference on Machine Learning, pp. 3732 3747. PMLR, 2020.
Philippe Hansen-Estruce Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit gleaannin actor-critic method with diffusion policies. arXiv preprint 2xXi:2334410574 2023.
Longxiang He, Li Shen, Junbo Tan, and Xueqian Wang. AlignIQL: Policy Aligument in Implicit Q-Learning through Constrained Optimization. CoRR, abs hts/2405.18187, 2024. URL https://doi.org:: /00/:/// 48550/
arXiv 2818.18187
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In Proceedings of NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL hnttss//oppmorriw.wntfftoo
Yd=
13

---

Published in Transactions on Machine Learning Research (12/2025)
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6885 2020.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Vineet Jain, Tara Akhound-Sadegh, and Siamak Ravanbakhsh. Sampling from energy-based policies using diffusion. In Proceedings of Reinforeement Learning Conference, 2025.
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. International Conference on Machine Learning, 2022.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generalive models. Advances in Newral Information Processing Systems, 35:26565 26577, 2022.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ODE trajectory diffusion. In Proceedings of The Twelfth International Conference on Learning Representa- tions, 2024. URL
Tya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learning. In Proceedings of International Conference on Learning Representations, 2022. URL https://openrevies: het/forumm2:d:68n2222222
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline rein- forcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv, abs/2005.01643 2020.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pp. 20035 20064. PMLR, 2023.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning (23/07/2023- 29/ 29/07/2023, Honolulu, Hawait), 2023.
Haofei Lu, Dongqi Han, Yifei Shen, and Dongsheng Li. What makes a good diffusion planner for decision making? In Proceedings of The Thirteenth International Conference on Learning Representations, 2025.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. Tn
Proceedings of International conference on machine learning, pp. 8162 8171. PMLR, 2021.
Erik Nijkamp, Mitch Hill, Song- Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run meme Loward energy-based model. Advances in Neural Information Processing Systems, 32, 2019.
Kwanyoung Park and Youngwoon Lec. Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning In Proceedings of The Thirleenth International Conference on Learning Representations, 2025.
Decpak Pathak, Parsa Mahmoudich, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shel- hamer, Jitendra Malik, Alexci A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEFE conference on computer vision and poltern recognition workshops, pp. 2053, 2018.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations (ICLR 2023), 2023.
William Peebles and Saining Xic. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVP international conference compuler vision, pp. 4195 3195-4205, 2023.
Michael Psenka, Alejandro Escontrela, Picter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. In Proceedings of the 4 1sst. International Conference on Machine Learning. pp. 41163-41182, 2024.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robolics: Science and Systems, 2018.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479 36494, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonoquilibrium thermodynamics. In Proceedings of International conference on machine learning, pp. 2256 2265. PMIRR, 2015.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In Proceedings of The Twelfth International Conference on Learning Representations, 2024. URL https::/openrevieer het:forrumidd-WWaayyBBBBa
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of International Conference on Learning Representations,
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proceedings of Inler- national Conference Machine Learning, 2023. URL https://epp..emmattcccoooooo htt.://tttcccctooooooooooooo//lo hwtp:oorr//////eeee
257280191
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-basod control. In Proceedings of IEEE/RSJ international conference on intelligent robots and systems, pp. 5026 5033. IEEE, 2012.
Guanquan Wang, Takuya Hiraoka, and Yoshimasa Tsuruoka. Planning with cousistency models for model- based offline reinforcement learning. Transactions on Machine Learning Research, 2024a.
Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Newral Information Processing Systems, 37:54183 54204, 2024b.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. In Proceedings of The Eleventh International Conference on Learning Representations, 2023. URL
Cheng-Fu Yang, Haoyang Xu, Te-Lin Wu, Xiaofeng Gao, Kai-Wei Chang, and Feng Gao. Planning in- in-painting: A diffusion-batt embodied task planning framework for environments under uncertainty. CoRR, 2023.
15

---

Published in Transactions on Machine Learning Research (12/2025)
A
Task Properties
7 provides a comprehensive summary of dataset sizes used in our experiments. All datasets are sourced from the latest release of the D4RL benchmark suite (Fu et al. 2020), encompassing four domains: Gym-MuuoCo, Maze2D, FrankaKitchen, and AntMaze.
Table
In the Gym-MuJoCo domain, we consider three data variants: medium-cxpert, medium, and medium-roplay. The medium-expert datasets comprise trajectories generated by a mixture of medium- and expert-level policies, thus incorporating both suboptimal and near-optimal actions. The medium datasets are collected exclusively using medium-performaane performance policies, and consequently, contain higher proportion of suboptimal behavior. The medium-roplay datasets represent replay buffers collected from medium-level agents during training, Teaturing highly diverse suboptimal trajectories and exploration noise. The Gym-MuuoCo domain consists of three standard environments:
Hopper involves a single-legged robot that must learn to hop forward without falling, presenting challenges in stability and balance.
HalfCheetah features a planar cheetah-like agent with a more complex morphology, requiring the agent to coordinate multiple joints to achieve high-speed locomotion.
Walker2d simulates a two-legged humanoid robot that must walk forward while maintaining upright posture, making it more prone to instability.
These tasks serve as standard benchmarks for evaluating the ability of offline algorithms to generalize from limited data and produce smooth, efficient motion without direct online interaction.
In the Maze2D domain, datasets are categorized according to maze complexity: umaze, medium, and large. Each variant reflects increasing navigation difficulty and state space dinensionality.
The FithhaKitchen domain includes two types of datasets: partial and mixed, both of which consist of undirected demonstrations. In the partial dataset, a subset of trajectories successfully accomplish the full task, enabling imitation learning algorithms to leverage these informative samples. Conversely, the mixed dataset contains only partial trajectories, with no complete demonstrations of the full task. This requires RL algorithms to generalize from sub-tri tabjjjjjessssis and effectively compose them into successful task completions.
For the AntMaze domain, use the same three maze configurations (umaze, medium, and large) in Maze2D. Three types of datasets are constructed: In the standard setting maulmaze-umaze-vil the ant is instructed to reach fixed goal from fixed start state. In the diverse datasets, both the goal and starting positions are randomly sampled, introducing high variability. In the play datasets, the ant is commanded to reach hind-picked waypoints in the maze, which may not coincide with the evaluation goal, and also begins from a curated set of slart locations. These variations are designed to assess the agent's ability to generalize under different levels of distributional shift and task ambiguity.
The Adroit Hand benchmark comprises four distinct manipulation scenarios, each instantiated with 30-DoF dexterous hand mounted on a freely moving arm:
Door. The agent must disengage latch with substantial dry friction and swing the door until it contacts the stopper. No explicit latch state is provided; the agent infers its dynamics solely through interaction. The door's initial pose is randomized across trials.
Hammer. The hand picks up a hammer and drives nail of variable location into board. The nail- subject to dry friction resisting up to 15N must be fully embedded for success.
Pen. With the wrist fixed, the agent reorients a blue pen so that its pose matches a randomly placed green target within a prescribed angular tolerance.
Relocate. The agent transports a blue sphere to a green target whose position, along with the sphere's start pose, is uniformly randomized throughout the workspace; success is declared once the sphere lies within e-ball of the target.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Domain
Task Name
Samples
hopper-me
2 106
hopper-m
106
hopper-mr
402000
halfcheetah-me
2 x 106 106
Gym-MuJoCo
halfchectah-m
hallcheetah-mr
202000
walker-me
2 10 106
walker-m
walker-mr
302000
maze2d-umaze
106
Maze2D
mazc2d-modium
106
maze2d-large
4 x 106
kitchen-mixed
136950
FrankaKitchen
kitchen-partial
136950
autmazo-medhummpplay
106
antmwwe-mediu--ddives.
106
hatmaze-large-payy
106
Aate
antmazo-largo-divesse
106
Table 7: Size for each dataset is provided. The number of samples indicates the total count of environment transitions recorded in the dataset (Fu al. 2020
B Additional Ablation study
Effect of applying CTM. To complement the controlled comparisons reported the main paper (Table 6 we provide additional results the Kitchen domain in this appendix. As in the Maze2D experiments, we evaluate three variants-CTP CP-improved, and DD-improved- identical architectural and algorith- mic seltings.
The results, summarized in Table S show that CTP consistently outperforms the other two variants in terms of average return on the Kitchen task. Furthermore, CTP achieves comparable or higher performance while requiring fewer denoising steps, demonstrating improved inference elliciency this more complex, high-dimensional control domain.
Table 8: Average normalized scores and inference time across different denoising steps N for CTP, CP- improved, and DD-improved in Kitchen environments.
Method
N
Average Score
Inference time (ms)
Kitchen- Partial
Kitchen-Mixod
44.8 + 0.9
52.7 2.1
9.7 0.3
СТР
71 1.7
64.8 + 1.0
11.3 0.3
91.2 1.0
74.5 = 0.3
13.3 + 0.3
$5.8
66.8 = 1.1
9.7 0.3
CP-improved
2
66.3 1.1
69 0.8
11.2 0.3
3
1.
14.3 1.0
13.2 0.4
0.0
23.8
9.6 0.3
DD-improved
5
5..1
52.1 + 1.5
24.1 + 0.4
10
80.1 1.2
66.5 + 1.0
40.0 + 0.3
Effect of the M-stride Trajectory Representation. We investigate the impact of the stride-based trajectory representation, where only every M-th state is sampled from the original sequence. We compare
17

---

Published in Transactions on Machine Learning Research (12/2025)
Effect of stride Trajectory Representation
Effect of Value based Trajectory Selection
stride
Normslired Retum
150
Normalized Return
150
100
75
50
25
MazeZD-Umaze
MazesD-Large
MazuaD-Umeee
Maz42D Medium
Maze20 Large
(a) Effect of M-stride trajectory representation.
(b) Effect of value-based trajectory selection.
Figure 5: The ablation study on (a) M-stride trajectory representation and (b) valuo-based trajectory seloc- tion across three Maze2D environments.
our original CTP (M 5, H 16) with CTP (no stride, H/ 80), both of which correspond to an effective lookahead of 80 steps.
Across all Maze2D tasks, CTP (no stride) achieves the best overall performance, followed by CTP (M = 5). These results highlight that introducing the M-stride representation slightly reduces performance but substantially improves computational efficiency, as it reduces the effective planning steps (16 vs. 80). Thus, the stride formulation offers a practical trade-off between accuracy and efficiency.
Quantitative results are shown in Figure 5a
Effect of Value-based Trajectory Selection. Next, evaluate the influence of the value network based selection used during inference. We compare CTP with filtering versus CTP without filtering, which uses classifier-free guidance for trajectory generation.
We further include the CP baseline, which also employs classifier-fres guidance by default, for comparison.
Results show that CTP with filtering achieves higher returns across all Maze2D variants, followed by CTP without filtering and CP. The results are shown in Figure 5b This suggests that: (i) under the classifier-free guidance mechanism, CTP consistently outperforms CP, demonstrating the inherent advantage of CTM in trajectory optimization; and (ii) value-based selection further improves planning quality by aligning trajectory choice with predicted returns, providing a more stable and reward-consistent mechanism than classifier-baase guidance.
C Implementation Details
This section outlines the architectural and training details used throughout our experiments.
CTM Training.
This is the Kim et al. 2024), included here for completeness.
distillation algorithm for consistency trajectory models (Song
et al.
2021
We use Diffusion Transformer blocks with adaL.-Zero architecture PPebles & Xie 2023 as the network backbone for the diffusion model and CTM.
Model Architecture.
Transformer Depth. We Transformer depth of 2 in all the MuJoCo tasks, Kitchen tasks, Maze2D lasks, and Adroit tasks, 8 in AntMaze-Meedium tasks, 12 in AntMaze-Large tasks.
Training Hyperparameters. The teacher model (EDM) trained using a learning rate of 2 10m4, while the student model (CTM) is trained with smaller learning rate of 8 x 10-6, Both models optimized
18

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm 2 Consistency Trajectory Model's Training
1: repeat
2:
Sample (T(T) == (sk, Mk+ 8K+(H-1)M) from data distribution
3:
Sample E ~ 10,1))
Sample & [c, [x]: w [0, u E (w,t)
Calculate x X + Le
Calculate SOLVER t, Φ) 8
Update 0 < 0
3 C(,, 11)
Update 17 +- 1 उn LGAN(0, m)
until converged
using the Adam optimizer with a batch size of 128 for a total of x 10th training steps. Additionally, we train the inverse dynamics model and the critic using a lcarning rate of x 10-4, also with the Adam optimizer.
Inference Settings. During inference, we set the number of sampling steps N 2 for CTP procedure.
Planning Horizon and Stride. We set the planning horizon H and stride M as follows:
MuJoCo tasks: H = 4, M =1
Kitchen tasks: H 32, M 4
Maze2D tasks: H 32, M 15
AntMaze tasks: H 40, M 25
Adroit tasks: H 32, .M =
These hyperparameters were selected based on task complexity and temporal resolution, with longer planning horizons and strides for more complex, long-horizon tasks such as AntM and Kitchen.
19


Reviews:
[{"review_id": "yYsIZQbODb", "reviewer": "Reviewer_oZMQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Consistency Trajectory Planning (CTP), a model-based offline RL algorithm that leverages Consistency Trajectory Models (CTMs) for trajectory optimization. The authors report strong performance and efficiency on diverse benchmarks, especially in long-horizon, goal-conditioned tasks. However, the method appears to be a relatively direct application of CTMs to prior work, and its advantages over existing approaches are not clearly shown.", "claims_and_evidence_yn": "No", "claims_explanation": "- Claim \"CTP is highly efficient, achieving over 120× speedup in inference time.\" \n\nThe baseline used, Diffuser, is known for its high computational cost and not a state-of-the-art; a comparison against state-of-the-art methods is needed for a meaningful assessment of efficiency.\n\n- Claim \"CTP consistently outperforms existing diffusion-based planning methods\"\n\nTable 1 shows D-QL has better performance and D-QL is notably absent from comparisons in Tables 2, 4, and 5. An explanation for its omission would help to fully support the claim of consistent outperformance.", "audience_yn": "Yes", "audience_explanation": "The application of consistency models to RL planning is an interesting direction, and RL planning remains a popular topic in machine learning.", "requested_changes": "1. The inference speed comparison should be extended to include more state-of-the-art (SOTA) algorithms, especially methods that are claimed to be more computationally efficient than Diffuser.\n2. Add results for D-QL (and maybe also other omitted methods) into all comparative tables, or provide a clear justification for their exclusion to ensure consistent evaluation.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "icdpuWcs2j", "reviewer": "Reviewer_eRuK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a model-based offline reinforcement learning method called Consistency Trajectory Planning (CTP) that generates trajectory segments conditioned on the current state for planning. The main contribution of this work is to use a Consistency Trajectory Model (CTM), instead of diffusion models as done in prior work, to reduce the high computational cost of iterative denoising. CTM allows fast denoising using fewer denoising steps while maintaining good sample quality. The proposed method, CTP, involves training a CTM over state trajectory segments, selecting the best plan based on a learned value function, and then using an inverse dynamics model to produce an action given two consecutive states from the generated trajectory. Experiments on D4RL locomotion, Maze2D, AntMaze, Kitchen, and Adroit tasks show that CTP achieves competitive performance compared to existing methods while using much fewer function calls (and hence lower inference time) compared to a diffusion-based method.", "claims_and_evidence_yn": "No", "claims_explanation": "1. The main empirical claim seems to be that CTP matches or outperforms existing diffusion-based methods. There are three design choices in CTP that differ from existing methods, particularly diffuser and DD, which seem to be the main point of comparison.\n    - Using a CTM instead of a diffusion model to generate trajectories.\n    - Generating multiple samples from the model and filtering them based on a learned value function, as opposed to using classifier or classifier-free guidance.\n    - Using jump trajectories that produce states every $M$ timesteps, giving it an effective horizon of $H \\times M$.\n\nThe focus of this work is clearly on incorporating CTM into the pipeline, but there are no experiments that control for the other two factors. The other two aspects, i.e., filtering multiple samples and generating every $M$th state, can be easily applied to diffusion-based methods as well. Without such experiments, it is not possible to determine whether using CTM alone is enough to match the performance of diffusion-based methods without the other modifications. It has been shown in previous work [2] that generating multiple samples and filtering seems to generally be better (this relevant paper should also be cited), and I suspect having a longer horizon of $M \\times H$ could be an important factor in tasks like Maze2D and AntMaze.\n\n2. The entirety of Sections 2 and 3 is very similar to the discussion in [1], apart from some light rephrasing and rewording. The authors should also clearly articulate what separates their method from [1] since there are quite a few similarities.\n\n3. There are some parts of the introduction that do not have supporting citations or do not seem to apply to CTP.\n    - “As a result, in practice, the multistep sampling procedure of CM … accumulates errors particularly as the number of function evaluations increases.” This statement is provided as is without supporting evidence or citations. This is important because using CTM instead of CM seems to be the key difference from [1].\n    - The introduction and several places in the paper mention “single-step sampling” as one strength of CTP, but in practice, the authors use different number of denoising steps depending on task complexity. Therefore, this phrase should be changed throughout the paper.\n\n*[1] Wang, Guanquan, Takuya Hiraoka, and Yoshimasa Tsuruoka. \"Planning with consistency models for model-based offline reinforcement learning.\" Transactions on Machine Learning Research (2024).*\n\n*[2] Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J.G. and Levine, S., 2023. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573.*", "audience_yn": "Yes", "audience_explanation": "There is quite a lot of interest in applying generative models to RL. This work is a step towards making these methods more viable by reducing the inference compute required, which is a notable limitation of existing methods. I believe the problem addressed in this paper and this particular approach is of interest to a sub-section of the community.", "requested_changes": "1. Please provide ablation studies on CTP to control for factors other than using a CTM instead of a diffusion model. Alternatively, the value network-based selection of trajectories and sampling every $M$th state in the sequence can be applied to all of the diffusion trajectory methods, including diffuser, DD, and CP.\n\n2. Please rewrite Section 2 and 3 substantially to include original content. Also, currently the background section discusses CM but not CTM, which is puzzling the latter is relevant to this work but the former is not. Without any background discussion on CTM, it is confusing to read about the method in Section 4 as teacher and student networks are mentioned without any context.\n\n3. Most of Section 4 is related to general CTM training, and is not specific to CTP. I suggest the authors move the discussion on CTM to the background section, and reserve Section 4 only for CTP-specific discussion. Also, currently the description of CTM and its losses is too verbose, it can be made much shorter since it is part of existing work.\n\n4. Please clearly differentiate CTP from the closely related existing work CP [1]. It is currently included as a baseline, but there is no discussion of this work in the related section or the experiment section.\n\n5. The related work section can be expanded to include many more existing works, including seminal work, on diffusion models applied to RL. There is a very rich literature in this area and the paper does not provide enough context to an uninformed reader. A few important works that may be included are [2,3,4,5,6].\n\n*[1] Wang, Guanquan, Takuya Hiraoka, and Yoshimasa Tsuruoka. \"Planning with consistency models for model-based offline reinforcement learning.\" Transactions on Machine Learning Research (2024).*\n\n*[2] Chi, Cheng, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. \"Diffusion policy: Visuomotor policy learning via action \ndiffusion.\" The International Journal of Robotics Research (2023): 02783649241273668.*\n\n*[3] Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J.G. and Levine, S., 2023. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573.*\n\n*[4] Psenka, Michael, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. \"Learning a diffusion model policy from rewards via Q-score matching.\" In Proceedings of the 41st International Conference on Machine Learning, pp. 41163-41182. 2024.*\n\n*[5] Jain, Vineet, Tara Akhound-Sadegh, and Siamak Ravanbakhsh. \"Sampling from Energy-based Policies using Diffusion.\" In Reinforcement Learning Conference, 2025.*\n\n*[6] Wang, Yinuo, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang et al. \"Diffusion actor-critic with entropy regulator.\" Advances in Neural Information Processing Systems 37 (2024): 54183-54204.*", "broader_impact_concerns": "This work does have any ethical considerations that need to be addressed, and does not require a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SHA2dY71sU", "reviewer": "Reviewer_nXd1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces **Consistency Trajectory Planning (CTP)**, an offline model-based RL planner that replaces the iterative denoising process of diffusion planners with a single- or two-step **Consistency Trajectory Model (CTM)** distilled from a score-based teacher. CTP rapidly samples candidate trajectories, ranks them with a learned return critic, and executes the first action via an inverse‑dynamics model.  Experiments on 27 D4RL tasks show comparable or superior returns to diffusion baselines while reducing planning latency by ≈120× on Walker2d.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. Bridges recent CTM advances with offline RL planning, addressing the pressing speed bottleneck of diffusion planners.\n\n2.  Evaluated on four domains (MuJoCo, Maze2D, AntMaze, Kitchen) with 150 seeds per task; wall‑clock comparisons included.\n\n3. Distillation loss combines global/local consistency, DSM regularization, and GAN loss to mitigate degradation.\n\n4. One‑shot (or two‑shot) sampling simplifies deployment.", "audience_yn": "Yes", "audience_explanation": "**CTP** connects several different research sub-areas in machining learning -- RL, generative model, planning... It is of great interest to different researchers.", "requested_changes": "1. CTP underperforms model‑free Diffusion‑QL on 6/9 locomotion tasks; the cause is not analyzed. So please Analyze the MuJoCo performance gap; consider increasing candidate count *N* or adjusting critic training.\n\n2.  Latency measured only on Walker2d; large maze tasks would better showcase the claimed scalability. Report latency on AntMaze‑Large and Kitchen tasks.\n\n3. Lacks convergence or error‑propagation analysis for CTM sampling.\n \n4. Header artefacts, small fonts in Figure 2. Fix presentation artefacts and enlarge figure fonts.\n\n5. It would be better if the authors could release code and pretrained CTM weights.", "broader_impact_concerns": "No direct ethical concerns.  Faster RL planners could accelerate deployment in safety‑critical systems.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)