Planning with Consistency Models for Model-Based Offline Reinforcement Learning
Authors: Guanquan Wang
Takuya Hiraoka
Yoshimasa Tsuruoka
Venue: Accepted by TMLR ()

Abstract: This paper introduces consistency models to the problem of sequential decision-making. Previous work applying diffusion models to planning within a model-based reinforcement learning framework often struggles with high computational cost during the inference process, primarily due to their reliance on iterative reverse diffusion processes. Consistency models, known for their computational efficiency, have already shown promise in reinforcement learning within the actor-critic algorithm. Therefore, we combine guided consistency distillation with a continuous-time diffusion model in the framework of Decision Diffuser. Our approach, named Consistency Planning, combines the robust planning capabilities of diffusion models with the speed of consistency models. We validate our method on Gym tasks in the D4RL framework, demonstrating that, when compared to its diffusion model counterparts, our method achieves more than a 12-fold increase in speed without any loss in performance.

Published in Transactions on Machine Learning Research (12/2024)
Planning with Consistency Models for Model-Based Offline Reinforcement Learning
Guanquan Wang
Deportment of Information and Communication Engineering
The University of Tokyo
Takuya Hiraoka
takugaa-h &nncc. com
NEC Corporation, Tokyo, Japan
Yoshimasa Tsuruoka
ac.jp
Department of Information and Communication Engineering The University of Tokyo
Reviewed on OpenReview: https:
// openreview. net/ forum? id= TuyCCzfty3
Abstract
This paper introduces consistency models to the problem of sequential decision-making Previons work applying diffusion models to planning within a model-based reinforcement learning framework often struggles with high computational cost during the inference pro- cess, primarily due their reliance on iterative reverse diffusion processes. Consistency models, known for their computational efficiency, have alrcady shown promisc in reinforco- ment learning within the actor-critic algorithm. Therefore, combine guided consistency distiilation with a continuous-time diffusion model in the framework of Decision Diffuser. Our approach, named Consistency Planning, combines the robust planning capabilities of diffusion models with the speed of consistency models. We validate our method on Gym lasks in the D4RL framework, demonstrating that, when compared its diffusion model counterparts, our method achieves more than a 12-fold increase speed without any loss in performance.
1 Introduction
In recent years, significant strides have been made in high-resolution image generation through the advance- ment of diffusion-based generative models. Similarly, in offline reinforcement learning (RL) scttings, deriving effective policies from pre-existing offline datasets can be simplified to the task of developing probabilistic model for trajectory prediction, area where diffusion- based generative models have proven to be highly successful. Existing models such as Diffuser (Janner et al.) (022)) and Decision Diffuser (Ajay et al. 2022) underscore the efficacy of applying diffusion models to planning within model-based RL frameworks. Diffuser, a diffusion model trained on the trajectories in offline datasets, and then a separate classifier model trainod to predict the cumulative rewards of trajectory samples. During the inference process, the diffusion model, combined with classifier guidance, employed to sample trajectories with high returns. Likewise, Decision Diffuser introduces a conditional diffusion model with state sequences input, utilizing the return as conditioning variable for classificr-free guidance during sampling. Moreover, by incorporating only the state sequence excluding the action sequence Decision Diffuser trains an extra inverse dynamic model to infer actions.
Parallel these developments, diffusion models have been adapted model-frec reinforcement learning scenarios, as illustrated by Diffusion-Q1 (Wang al. (022) and further enhanced in the efficient diffusion policy (EDP) (Kang et al. 2024). Diffusion-QT, utilizing denoising diffusion probabilistic model (DDPM)

---

Published in Transactions on Machine Learning Research (12/2024)
(Ho et al. 2020,, frames the diffusion model policy representation, conditioned states with actions outputs. It integrates Q-learning guidance into the reverse diffussin process to seek optimal actions. Despite its advancements, Diffusion-Q2 faces limitations in computational efficiency and its exclusive application within TD3 algorithms (Fujimoto & Gu 2021). EDP addresses these issucs by introducing an action approximation trick during training, applying the DPM-solver, and approximating policy likelihood via the evidence lower bound the DDPM to overcome the limitations of Diffusion-QI.
Although the integration of diffusion models within both model-based and model-free frameworks in the offline RL setting has been extensively explored and enhanced, a significant challenge remains in their application, particularly in real-time decision-making contexts. This challenge stems from the diffusion models' reliance on iterative sampling processes, which can be computationally intensive and slow, thus restricting their use in scenarios that require rapid inference. For instance, in robot arm control (Chi et al, 2023), standard diffusion-basedi control can only make decisions around 10Hz. However, this is insufficient for tasks requiring agile motion planning at 20Hz (Smith et al., 2023). 30Hz (Peng et al.) 2020), or even higher.
Recent endeavors by Song et al. (0023) have introduced consistency models, novel class of generative models that significantly enhance computational efficiency without sacrificing the expressiveness and flexibility that make diffusion models appealing for reinforcement learning.
In the model-free RL domain, consistency models have demonstrated promising results as policy represen- tations, particularly in offline and offine-to-online RL settings (Ding & Jin 2023). These developments un- derscore consistency models' capability to effectively navigate the challenges of learning from fixed datasets, indicating their potential to achieve performance comparable to diffusion-based approaches, while maintain- ing higher computational efficiency.
However, in offline RL settings, model-free methods using Q-network face challenges due to overestimated Q-values for out-of-distribution actions (Kumar et al., 2020 Levine et al.) 2020). In the context of online RL, the problem is self-correcting as the policy interacts with the environment: an action perceived as favorable might receive a low reward, thus adjusting the policy. However, in offline RL, such corrections are not readily achievable, which often leads the learned Q-function to often guide the diffusion model towards potentially sub-optimal actions. Therefore, given the computational efficiency of consistency models and the proven effect iveness of difffusion models in trajectory prediction, this paper aims to explore how consistency models can augment model-based RL with classifier-free guidance in an offline setting, thereby bypassing the necessity of learning a Q-function by conditioning the consistency models on returns.
The goal of this paper is bridge this gap by proposing a novel approach that merges the computational efficiency of consistency models with the planning capabilities inherent in Decision Diffuser. By integrat- ing consistency models into the trajectory optimization process, we aim to leverage their computational advantages to enhance the speed of planning. Our experiments, conducted in offline RL settings, embed conditional consistency model in the Decision Diffuser algorithm, evaluating with consistency distillation methods. Specifically, the Consistency Model employs guided consistency distillation from a score-based
diffusion model [arras et al., 2022 Ho & Salimans 2022 datasets.
Luo et al. 2023 pretrained on offline trajectory
In summary, our contribution is proposing Consistency Planning, novel offline RL algorithm that extends the applicability of consistency models model-based RI. We evaluate Consistency Planning D4RL benchmark tasks (Fu et al. 2020) for offline RL, demonstrating that this method can achieve performance comparable to its diffusion model counterparts across the majority of tasks, while offering notably faster sampling process.
2
Related Work
2.1 Diffusion Models
Diffusion models have emerged as a powerful approach for generating high-quality image and text data, demonstrated by previous studies (Saharia al. 2022 Nichol & Dhariwal 2021). The data sampling
2

---

Published in Transactions on Machine Learning Research (12/2024)
process is formulated as iterative denoising procedure, introduced by Sohh-Diccstein et al. 2015 and further developed by (Ho et al. (2020). Parameterizing the gradients of the data distribution serves as an alternative interpretation of this denoising procedure, aiming to optimize the score matching objective, as elucidated by Hyyvärnnnn & Dayan 2005). This positions the approach within the domain of Energy-Basse
Models, as evidenced by the contributions of Du & Mordatch (2019); Nijkamp et al. (2019) and Grathwohl
ct al. 2020).
Prior work Nichol & Dhariwal 2021 has implemented classifier to enable the generation of images based conditional information (e.g., text), ) which is called classifier guidance. However, more recent studies (Ho & Salimans| 2022), propose classifier-free guidance, which relies on the gradients from an implicit classifier, derived from the score function differences between conditional and unconditional models. This approach has proven to enhance the quality of conditional samples compared to classifier guidance methods. These advancements predominantly focus on text and image generation.
2.2 Diffusion Models in Reinforcement Learning
Diffusion models offer a versatile approach for data augmentation reinforcement learning. SynthER (Lu et al. 2024) employs unguided diffusion models to enhance both offline and online RL datasets, subsequently utilized by model-free off- off-policy algorithms. Although this approach boosts performance, SynthER's reliance unguided diffusion to approximate the behavior distribution faces challenges due to distributional shift. Similarly, MTDiff (He et al. 2024 implements unguided data generation in multitask environments.
Additionally, diffusion models have been adapted for training world models. For instance, Alonso et al. (2023) use diffusion to train world models, achieving precise predictions of future observations. However, this method does not model entire trajectories, leading compounded errors and lack of policy guidance. In related effort, Rigter et al. (2023) integrate policy guidance to enhance diffusion world model in online RL. Jackson et al. (2024) concentrate on offline RL, providing theoretical framework and rationale for the trajectory distribution shaped by policy guidance.
Diffusion models have also been adapted for policy representation in RI,, capturing the multi-modal distri- butions in offline datasots. Specifically, Diffusion-QL (Wang ct al. 2022), applics the diffusion model within the framework of both Q-learning and Behavior Clouing (BC) for policy representation. However, the main limitation of Diffusion- QL that it demonstrates computational inefficiency due to the necessity of process- ing both forward and backward through the entire Markov chain during training. To alleviate these issues, Kang et al. (2024) introduce action approximation, eliminating the need to execute the denoising process during the training process.
Diffusion models have been employed in recent studies for human behavior imitation learning PPearee et al. 2023) and trajectory generation in offline RL. Trajectories that include states and actions are generated by Diffuser (Janner et al., (022), using an unconditional diffusion model, guided by a reward function trained on noisy state-action pairs. Decision Diffuser (Ajay al.] (002)) models the trajectories with the dataset using a unified, conditional generative model, avoiding separate training classifier for reward functions.
3 Preliminary
3.1 Reinforcement Learning Problem Setting
The sequential decision-making problem is defined as a Markov decision process (MDP): M = A, do) where S and A are the state space and the action space respectively, P S A + represents the transition function, R :: S Ax + R denotes the reward function, 7 E [0,1) 1) is the discount factor, do is the initial state distribution. The goal of RL is learn policy no (als) to maximize the expected sum of discounted rewards E. [E (sk, ax)].
3

---

Published in Transactions on Machine Learning Research (12/2024)
3.2 Consistency Models
Diffusion models operate by introducing Gaussian perlurbations to transform data into noise, followed by generating data samples through a series of sequential denoising steps. [Song et al. (2020) introduce a stochas- tic differential equation (SDE) framework that ensures the maintenance of the desired distribution as sample x evolves over time. The consistency models proposed by Song et al. (2023) recover the original data sam- ple by solving a corresponding probability flow ordinary differential equation (ODE): dr -IV log, Pt(x), where px(x) Pdota (x) @ (1), Pdato represents the original data distribution, E [0, 2] is the time period. The data generation process in this framework reverses along the trajectory ( of the ODE, starting from random initial samples îT ~ N (0, 10, where e is a minimal constant close to to address numerical issues at the boundary.
To accelerate the sampling process diffusion models, the consistency model significantly reduces the number of steps required for sampling compared to the original diffusion model, without substantially compromising the model's performance. This is achieved by approximating parameterized consistency function, fo (z), Tee which maps noisy sample at step & back to the original sample Ie.-
This approach differs from the diffusion model, which utilizes a step-by-step denoising function (x--1 2x), for the reverse diffusion process. Slightly different from the original consistency model, this paper focuses on conditional distribution, so the consistency function modified to S' (Tre c), where denotes the condition variable.
4 Planning with Consistency Model
This paper explores the integration of consistency models, which are trained by distillation from a pre-trained diffusion model, into the planning architecture of Decision Diffuser. The original consistency models draw from the principles of based diffusion models (Song et al. 2020 Karras et al. 2022), making direct distillation from the discrete-time model used in Decision Diffuser ineffective. In the following, we discuss how we use consistency models for the trajectory optimization process. Section 4.1 details the training process of the diffusion model, followed by an explanation in Section 4.2 of how guided consistency distillation is applied, and how the consistency model is integrated with the Decision Diffuser framework during inference.
4.1 Diffusion Model Training
Trajectory reprosentation. As outlined by Ajay et al. (0022) the diffusion process encompasses only the state transitions as described by
De.t1 (T) (sk, Sk+1 ................................ ; Sk+H-1)
(1)
In this notation, k indicates the timestep of a state within trajectory T, H represents the planning horizon, and f++1 is the timestep in the diffusion sequence. Consequently, ze44, (T) is defined as noisy sequence of states, represented as two-dimensional array where each column corresponds to a different limestep of the trajectory, In training process, the sub-sequence t; E [e, follows the Karras boundary schedule (Karras
ct al. 2022):
1/0 + i i-1
-c10))
t,
(2)
where 6 0.002, tN = 80, and 7.
Acting with inverse dynamics. To derive actions from the states generated by the diffusion model, we employ an inverse dynamics model (Agrawal et al. 2016 Pathak et al.] 2018 denoted as h1p trained
using the same dataset as the diffusion model. Actions can be obtained via the inverse dynamics model by extracting the consecutive state sy and 9441 at diffusion timestep to:
Ok h (8)
(3)
4

---

Published in Transactions on Machine Learning Research (12/2024)
Combined training of diffusion models and inverse dynamics. Given dataset D consisting of trajectories, each labeled with its respective return, the combined training of the diffusion model (denoted by D&) and the inverse dynamics model is conducted using the following loss:
[ :p) Poa Bern((p Beerrr)
(t)) 3)c(r) + 30, -
(4)
|| (s, 27
+ (s yDD
where Ptrain is a lormnormal distribution using the design choice from Karras et al. (2022 8 is sampled from Bernoulli distribution with probability p. Namely, the condition information c(T) ignored with probability p. which is manifested by the condition information being an empty set 0.
We employ returns R(r) under trajectories as the conditioning information c(T), normalized such that R(T) E []. We map it into latent variable E g using multi-layer perceptron. In cases where R(T) = 0, the components of с are set to zero. During the inference time, sampling trajectories with high returns corresponds to conditioning on R(T) =.
4.2 Consistency Distillation
Guided Consistency Distillation. Incorporating classificr-free guidance essential for synthesizing high- return trajectories. Considering the compulational demands and potential for error accumulation associated with two-stage distillation methods (Meng et al. 2023), opt for one-stage guided distillation approach
as proposed by Luo et al. (2023).
Algorithm Consistency Distillation with guidance
1: Input: dataset D, intial consistency model parameter 9, learning rate v,, ODE solver & ( distance metric (,;,,, EMA rate μ4, noise schedule l,, guidance schedule [4minn Wmar).
Initialize target consistency model 0- +
repeat
U[1, - -1] and guidance U [Wmmin Wmaz] Sample trajectory and condition (I, ~ D, 1
5: Sample noised trajectory Th.s ~ NNx (a+11)
1/.w +- Ttat1 + [(w + (,
( ((b1;1)))) Φ)]
// Classifier-free guidance
7:
C(0, 00000:000 - d (,
(0t w. C, tn)
// Calculate consistency distillation loss
8:
0- V 0 ;0)
// Update consistency model parameter // Update target consistency model paramcter
8- +- stopgrad + (1 -))
10: until convergence
The consistency function fe (Ir,w, c) + xo parameterized to transform state x, at time t directly into the original state zg- We parameterize fo in the same way [Son et al. (2023), except that we consider the influences of guidance scale and conditioning variable
fo fo(x, w, e, t) = (t)x + Cout (o,,,,)))
(5)
where Fg is free-form neural network with an output that matches the dimensionality of x, Caksp(t) = and (c) = 0 so that fo satisfies boundary condition fo(x, (it,,,,,,, c, x. During the distillation process, the guidance scale w and n are sampled uniformly from the intervals Homin,Wmas and {1. 11, 1}, respectively. The trajectory and returns tuple (æ, c) are sampled from the dataset. Then, s6."" is estimated by employing an ODE solver 4:
26.0 Dtass BJ [(w + (Tr (n ()
wdl 0, [n)1
(6)

---

Published in Transactions on Machine Learning Research (12/2024)
Finally, minimize the consistency distillation loss (Song et al. 2023 distillation:
Luo et al. 2023 used for guided
[(0, 0(0,0-00 = Eg,c,,n [d (f tnt1), for (o w c,),
(7)
where d is squared l2 distance d(z, v) = I|(x y)1l
The pseudo-code for guided consistency distillation adapted for trajectory generation is shown in Algorithm
Algorithm 2 Planning with Consistency Model
Input: consistency model fe, inverse dynamics hip guidance scale w, history length C, condition c, sequence of time points > ... fN-1, initial noise ær; fixed small positive number c. Initialize history h 4 Queue(length = C)2 0.
// Keep a history with a maximum length C
3: while not done do
4: Observe state 8; h.insert(s);
5: Initialize x(T) + fo (00,,,,,,, C, or ~ No(,T21) // Generate samples and evaluate the consistency
model in single step
for to N v-1c0 do
// Evaluate the consistency model in multiple steps // Ensure the trajectory aligns with the history
x(r)([, (eighh(h) +
8:
ât, (T) + x(T) + -2,
9:
x(T) + fā,
10: end for
11: Extract state tuple (sk, Sk+1) from generated trajectory x(T)
12: Execute action 0ş hip (8k: 8k+1)
13:
end while
Consistency Model Inference. During the inference process, we first observe a state 8 in the environment and sample an initial trajectory Zy. Then, our consistency model, conditioned on returns c, guidance scale w and history of last states observed, iteratively predicts the denoised trajectories from the noisy inputs ét, (r) + z(T) + Vta 222 along the probability flow ODE trajectory at step n E [N], with Gaussian noise (O, T). For the single-step version of Consistency Inference, {t,,
0, {e, 7)- Finally, extract states (sg, 8k41) from denoised trajectory and get the action ag via our inverse dynamics model hg. The algorithm of Consistency Planning is provided in Algorithm Zadd visualized in Figure [ For the architecture and implementation details, please refer to Appendix.
Etx (T)
guided consistency distillation
8k
Sk-
5k
8k
$
returss
pre-trained diffusion madel
consistency model with guidance scale
x (T)
8k
Sk-1
8k H1
inverse dynamics
dx
Figure Consistency Planning. Given the current state 85, conditioning variable and guidance scale w, Consistency Planning generate sequence of future states with planning horizon H. Then the inverse dynamics model is used to extract and execute the action og from Sk and 8k+1

---

Published in Transactions on Machine Learning Research (12/2024)
5 Experiment
In our experiment section, we evaluate the performance of the Consistency Planning offiie RL settings and its computational efficacy in Section 5.1] and provide detailed ablation study to test the effectiveness of the inverse dynamic model he and the guidance scale w in Section 5.2
5.1 Offline Reinforcement Learning
The diffusion model, inverse dynamics model, and consistency model are trained using publicly available DARL datasets and subsequently evaluated Walker2d, and Maze2D, within the D4RL. benchmark suite (Fu et al. (020). These tasks are character- ized by continuous state and action spaces and are conducted under offline reinforcement learning settings. Details of the dataset size are provided in Appendix.
range of Gym tasks, including HalfCheetah, Hopper,
We compare the performance of our method with those of both behavior-cloning methods, i.e., Consistency- BC (C-BC) DDig & Jin 2023), Diffusion-BC (D-BC) (Wang al., 2022], actor-critic methods, i.e., Consistency- AC (C-AC) (Ding & Jin 2023). Diffusion-QL (D-QL) 4 (aag al. 2022) algorithms, and model-based methods, i.e., Ditfuser (Janner et al. 2022). Decision- (DD) AAay al. 2022 in Table For evaluation, results for our method correspond to the average over 150 planning sceds. By default, our consistency model applies the number of denoising steps N = with saturated performance on most tasks, while the diffusion policy uses N
(Wang et al., 2022). Diffuser and Decision Diffuser use N = 20 and
N 4040, respectively (Janner et al. 2022 Ajay ct al. 2022 For the performance metric, we use normalizod
average returns (Fu et al. 2020) for all tasks.
Table shows that although our method achieves slightly lower average score (82.2) than Diffusion- QL (87.9) and Consistency-AC (85.1), as model-based planning model, it outperforms its diffusion counterparts, i..., Diffuser (75.3) and Decision Diffuser (81.8), with the reduction of denoising steps in the inference stages.
However, is worth noting that our model exhibits a more pronounced performance gap in the half- cheetah environment compared to the hopper and walker2d environments compared with Diffiusion- QL and Co This can be attributed to several factors. While our model employs classifier free guidance, it lacks the value function estimation provided by Q-networks in Consistency-AC and Diffusion-QI. These Q-network-based methods explicitly optimize for cumulative rewards, allowing them to perform better in environments like half-cheetah, where subtle variations in action sequences can have significant impact on long-terp term returns. Moreover, half-cheetah presents a higher degree of control complexity and a more dynamic action space compared to hopper and walker2d. The need for procisc coordination among multiple joints makes it more challenging for generative models like ours, which rely purely on the consistency model and classífier-free guidance, to generate optimal action sequences. Tn contrast, Q-network-based methods are better equipped to navigate this complexity by explicitly evaluating state-action pairs in terms of their future rewards.
To validate the long-horizon planning capabilities of Consistency Planning, we conduct an evaluation in the Maze2D environment (Fu et al. 2020). where the task involves navigating to a specific goal location, with a reward of 1 assigned only upon reaching the goal. Since it requires hundreds of steps to reach the goal, even the most advanced model-free algorithms struggle with effective credit assignment and consistently reaching the goal. As shown in Table 2 Consistency Planning achieved scores excceding 100 across all maze sizes, indicating that it outperforms the reference expert policy.
To assess the computational efficiency of Consistency Planning and its diffusion model counterparts, Decision Diffuser, we conduct experiments to mcasure inference time (ms per sample) in the hopper-medium--xperr on our server. Both the consistency model and the diffusion model, as generalive models based on probability flow, have computational times that are directly dependent on the number of denoising steps N. The results in Figure g show that N = 2 for Sesiistnnc-PPlaniingg where each algorithm achieves its saturated performance in hopper-medium medium-expert.. The mean and standard deviation of results are calculated over five random seeds.
and N = 20 for Decision Diffuser, are the values
7

---

Published in Transactions on Machine Learning Research (12/2024)
Table
The average scores of vanilla. BC (with Gaussian), Diffuser, Decision Diffuser, Diffusion-BC.
Consistency-BC, Diffusion-QL, Consistency-AC and our method on D4RL Gym tasks are shown, with stan-
dard deviation reported for Consistency Planning. All results are quoted from Ding & Jin (2023) and Ajay
et al. 2022p.
Dataset
BC
Diffuser
DD
D-BC
C-BC
D-QL
C-AC
Ours
Halfcheetah-me
55.2
79.8
90.6
90.8
32.7
96.8
84.3
94.0 1.3
Hopper-me
52.5
107.2
111.8
107.6
90.6
111.1
100.4
107.5 + 1.8
Wdlker2d-me
107.5
108.4
108.8
108.9
110.4
110.1
110.4
109.8+0.5
Halfcheetah-m
42.6
44.2
49.1
45.4
31.0
51.1
69.1
46.8 = 1.2
Hopper-m
52.9
58.5
79.3
65.3
71.7
90.5
80.7
87.8 1.6
Walker-m
75.3
79.7
82.5
81.2
83.1
87.0
83.1
80.5 = 0.8
Halfcheetah-mr
36.6
42.2
39.3
41.7
34.4
47.8
58.7
40.6 0.9
Hopper-mr
18.1
96.8
100
67.9
99.7
101.3
99.7
97.8 = 0.8
Walker2d-mr
26.0
61.2
75
77.5
73.3
95.5
79.5
75.3 1.1
Average
51.9
75.3
81.8
76.3
69.7
87.9
85.1
82.2
Avg. Norm Score for "hopper-medium-eexpettvz task
100
Avg. Norm Score
80
60
40
20
Decision Diffuser Consistency Planning
15
20
N
Figure Comparison of average norm score vs. for Decision Diffuser and Consistency Planning on the task medium-cxpert--v2
Since the computational time for these models directly tied N, and the consistency model designed require fewer denoising steps achieve similar generative performance, claim our model has achieved more than 12-fold increase in speed without any loss in performance. More detailed information concerned the

---

Published in Transactions on Machine Learning Research (12/2024)
The performance of Consistency Planning, Diffuser, and previous model-free algorithms in the Maze2D environment, which tests long-horizon planning due to its sparse reward structure. Consistency
Table 2:
Planning's s performance is comparable to that of Diffuser, with diffusion steps N = 2. All results are derived from the data provided in Janner et al. (002)).
Dataset
MPPI CQL
IQL
Diffuser
Ours
Maze2D U-Maze
33.2
5.7
47.4
113.9 3.1
122.7 7 2.7
Maze2D Medium
10.2
5.0
34.9
121.5=2.7
121.4=4.1
Maze2D Large
5.1
12.5
58.6
123.0=6.4
119.5 + 7.5
Average
16.2
7.7
47.0
119.5
121.2
Table 3: Ablation study on inverse dynamics.
Hopper-* *
Consistency Planning w/o inverse dynamics
Ours
Med-Expert
105.9 + 1.7
110.4 + 2.4
Medium
78.3 = 2.4
88.7 + 2.2
Med-Replay
89.4 = 3.8
96.7 = 1.6
inference time v.s. N across the other hopper environments, halfcheetah-medcu-epppooe and walker2d- medium-expert-v2 shown Appendix.
5.2 Ablation Study
Inverse dynamics. To evaluate the role of inverse dynamics in our approach, we conducted a compara- tive analysis with Consistency Planning models that exclude the inverse dynamics component, where the trajectories consist of both state and action sequences. The performance results are presented in Table 3 Our findings indicate that incorporating inverse dynamics Icads to better performance across all three hop- per environments, consistently outperforming the variant without inverse dynamics. This demonstrates the effectiveness of inverse dynamics in improving the model's ability to generate action sequences, contributing enhanced planning performance.
Guidance Scale. In our experiments, we examine the impact of guidance scale w on the performance of our consistency models during planning. The guidance scale controls the weight of the guidance signal during the planning process, and its effect on performance can vary significantly depending on the characteristics of the offiine dataset. To assess the sensitivity of our model to w, conducted experiments across different types of datasets, including hopper medium-expert, hopper medium, and hopper medium-replay.
For the hopper medium-expert dataset, our results in Figure indicate that varying the guidance scale w does not lead to substantial differences in performance. This can be attributed to the relatively high-quality data the medium-expert dataset. As a result, the model performs well across a wide range of values, and the influence of the guidance signal is less pronouneed.
However, when applied to hopper medium and medium-replay datasets, observe marked difference in performance based on the choice of W. Specifically, larger guidance scales (e.g., 0.8) consistently result in better outcomes compared to smaller scales (c.g., w = 0). This is particularly evident in the medium setting, where the dataset. contains suboptimal trajectories alongside more desirable ones. In such cases, the stronger guidance provided by larger w helps the model better navigate the diverse quality of trajectories, ultimately Icading more consistent and improved performance. On the other hand, smaller w values may not provide sufficient signal strength guide the model away from suboptimal decisions, resulting in degraded performance.
9

---

Published in Transactions on Machine Learning Research (12/2024)
Avg.
Norm Score vs w for Hopper Environments with Standard Deviations
120
110
Avg. Norm Score
100
90
hopper-me
hopper-m
hopper-mr
0.4
Figure 3: Ablation study on different guidance scales across 3 hopper environments. Larger leads to
better sample quality.
6 Conclusion
By combining the score-based diffusion model proposed by Karras ct al. (2022 onc-stage guided distillation (Luo et al. 2023), and conditional model-based generative model for sequential decision making (Ajay et al. 2022), the consistency model in this paper achieves comparable performance in gym tasks with its diffusion model counterparts, Diffuser and Decision Diffuser, and obtains a significant speedup during inference in oflline seltings. While our work increases the inference speed compared with its diffusion models counterpart (Ajay et al., 2022), there still exists some potential challenges of applying the approach scenarios where quick, real-time decision making critical.
One key challenge is scalability and model complexity. As consistency models are applied to larger and more complex environments, such as autonomous driving or stock market. trading, the number of variables and possible future states grows exponentially. Managing this complexity in real-time, while ensuring consistent and reliable outcomes, presents a significant challenge without the use advanced computational techniques. Another challenge involves data availability. Effective real-time decision-making requires access to up-to-date and accurate data streams. In an online sctting, the Consistency Planning method requires completing an entire episode to collect the trajectory before updating the model, which can lead suboptimal results in dynamic environments. This data collection process introduces delays that may hinder the effectiveness of real-time applications.
Future work should include: 1) combining improved lechniques in training consistency models (Song & Dhariwal 2023), such as designing a changing weighting function and noise schedule more suitable for reinforcement learning scenarios; 2) combining the consistency inference process with changing guidance schedule (Ma al. 2023) to improve the quality of trajectory sampling; and 3) investigating online learning strategies to reduce delays and improve performance in real-time scenarios.
10

---

Published in Transactions on Machine Learning Research (12/2024)
References
Pulkit Agrawal, Ashvin V Nair, Picter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016.
Amurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is condi- tional generative modeling all you need for decision-makien making? arXiv preprint arXiv:2211.1663 2022.
Eloi Alonso, Adam Jelley, Anssi Kanervisto, and Tim Pearce. Diffusion world models. 2023.
Cheng Chi, Siyuan Feng, Yilum Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song, Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprinl arXiv:2303.04137, 2023.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXim:2309.1690 2023.
Yilun Du and Igor Mordatch. Implicit gencration and modeling with energy based models. Advances in Newral Information Processing Systems, 32, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural informalion processing systems, 34:20132 222-20145, 2021.
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In International Conference on Machine Learning, pp. 3732-3747. PMLR, 2020.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances in newral informalion processing systems, 36, 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXic:2207.1559 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Matthew Thomas Jackson, Michael Tryfan Matthews, Cong Lu, Benjamin Ellis, Shimon Whiteson, and Jakob Focrster. Policy-guided diffusion. arXiv preprint orXiv:2404.06333 2024.
Michael Jannor, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint ar arXiv:2205.00 09991, 2022.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Tero Karras, Mila Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basod generative models. Advances in Neural Information Processing Systems, 35:26565-26577 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Newral Information Processing Systems, 33:1179 13:1179-1191 1191, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Ofline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016 01600, 2020.
Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder, Synthetic experience replay. Advances in Neural Information Processing Systems, 36, 2024.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint Xxi::3300004975 2023.
Jiajun Ma, Tianyang Hu, Wenjia Wang, and Jiacheng Sun. Elucidating the design space of bussifier-guider diffusion generation. arXwv preprint Xrx:223310 1311, 2023.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognátion, pp. 14297-14306, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162 8171. PMER, 2021.
Erik Nijkamp, Mitch Hill, Song- Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcme toward energy-based model. Advances in Newral Information Processing Systems, 32, 2019.
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shel- hamer, Jitendra Malik, Alexci A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference computer vision and pollern recognilion workshops, pp. 2050 2053, 2018.
Tim Pearce, Tabish Rashid, Aussi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, al. Imitating human behaviour with diffusion models. Xrviv preprint or arXi:22301. 10677, 2023.
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.0078, 2020.
Marc Rigter, Jun Yamada, and Ingmar Posner. World models via policy-guided trajectory diffusion. arXiv preprint ArXi::2312 2. 08533, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffussin models with deep language understanding. Advances in neural information processing systems, 35:36479 36494, 2022.
Laura Smith, J Chase Kew, Tianyu Li, Linda Lau, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Learning and adapting agile locomotion skills by transferring experience. arXie preprint 098884 2023.
Jascha Siil-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256 2265. PMLR, 2015.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. orXiv preprint arXiv:2910.14189 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXio preprint arXiv:2011.13456, 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 0:X11:2303.01469, 2023.
arXiv preprinl
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXio preprint or Xrx:2220 06199, 2022.
12

---

Published in Transactions on Machine Learning Research (12/2024)
A
Appendix
Table is a full list of datasct size. We use the latest version of dataset in D4RL (Fu et al. 2020 Our experiments utilize D4RL datasets comprising three distinct data types for the Hopper, HallCheetah, and Walker environments: medium-expert, medium, and medium-replay. The medium-expert datasets contain trajectories generated through a combination of medium-level and expert-level policies, offering a blend of both optimal and suboptimal actions. The medium datascts are produced exclusively by medium-level policies and therefore exhibit greater proportion of suboptimal actions compared to the medium-expert datasets. The medium-replay datasets represent the replay buffer of medium-leved agent, encompassing diverse range of suboptimal actions along with exploration noise.
The results in Table 5 show the relationship between the computational time, denoised steps N and corre- sponding performance of Consistency Planning and Decision Diffuser on the task hopper-mddum-expertyyy The results in Table G show the inference time and corresponding performance of Consistency Planning and Decision Diffusuer in different datasets, i.e hopper-mr, hopper-m, halfcheetah-mo and walker2d-me with diffusion steps N = 100 for Decision Diffuser and N = 2 for Consistency Planning. The choice of diffusion steps in Table G depends on the default training hyperparamters in Ajay et al. (2022) which clains all tasks sulfer no performance loss with N 100. Each cell contains the mean and standard deviation over 5 ran- dom seeds in the same server. As demonstrated in Table 5 we achieved more than 12-fold increase in speed without any loss in performance with N-2 for Consisitency Planning and N=20 for Decision Diffuser.
Table 4: Size for each dataset is provided. The number of samples indicates the total count of euvironment transitions recorded in the dataset Fuu al. 2020
Dataset
Samples
Hopper-me
2 x 106
Hopper-m
106
Hopper-mr
402000
Halfcheetah-me
2 106
Halfcheetah-m
106
Halfcheetah-mr
202000
Walker-me
2 x 106
Walker-m
106
Walker-mr
302000
Maze2D U-Maze
106
Maze2D Medium
2 106
Maze2D Large
x 106
In the next section, we describe various architectural and
details:
We parameterize the diffusion model and consistency model with a temporal U-Net architecture, a neural network consisting of repeated convolutional residual blocks from Janner et al. (2022) with 2nd order Heun as ODE solver, the inverse dynamics hp using the structure of Ajay
(0022).
We train diffusion model using learning rate of le - 4 and batch size of 512 for 2e5 train steps with Adam optimizer.
We choose the probability p of removing the conditioning information to be 0.25.
We use N for consistency inference.
We use planning horizon H of 32, context length C of 8 in all tasks.
We use guidance scale Wmaz 1, Wyin O in guided consistency distillation.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Table 5: Comparison of computational time for Decision Diffuser and Consistency Planning on the task hoppe-mediun-expptt-22 (Ajay et al. 2022).
Method
Inference Time (ms per sample)
Avg. Norm Score
40
837.6 + 8.4
110.0 110.1 = 0.5
20
427.7 3.9
10
216.8 + 1.2
104.0 18.9
Decision Diffuser
107.3 0.5
7 20.2
44.5 0.3
12.6
23.1 0.4
11.1
40
752.1 + 2.0
112.2 1.5
20
331.9 1.9
+
10
167.3 0.9
110.8
Consistency Planning
80.58 0.7
111.3
33.2 +
108.7
16.8 0.2
2.71
Table 6: Comparison of inference time for Docision Diffuser and Consistency Planning on the different datasets with respective default diffusion steps (Ajay el al. 2022).
Method
Dataset
Inference Time (ms per sample)
Avg. Norm Score
Hopper-m
1954.3 + 5.4
79.1 + 3.6
Hopper-mr
1938.5 + 3.7
100.5 + 0.4
Decision Diffuser
Halfcheetah-me
1920.4 + 4.5
89.2 + 1.2
Walker2d-me
1899.9 + 7.8
107.5 + 2.1
Hopper-m
33.5 + 1.8
88.6 + 1.1
Hopper-mr
33.7
96.9 + 1.3
Consistency Planning
Halfcheetah-me
33.1 + 2.1
94.1 + 0.9
Walker2d-me
32.9 + 0.8
108.5
14


Reviews:
[{"review_id": "S9hDgV1rfJ", "reviewer": "Reviewer_mDYs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The speed improvements due to a consistency model are good.\n\nWeaknesses:\n- Given that speed of inference is one of the main highlighted contributions, it does not make sense that the paper only presents Figure 2 and concludes that there is a 12x speedup.\n- The paper appears to be very incomplete - the experiments are sparse, the authors don't discuss the choice of their D4RL domains (when would the method work, when would it not work), don't provide ablations to their (implicit) choices of hyperparameters.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the use of consistency models for model-based offline RL, which allows for faster inference, which is a problem with diffusion models due to their iterative sampling process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please add more details to experiments (see Weaknesses point 2 above).\n- Please define the metrics you have used.\n- Please provide full details to reproduce the experiments (hyperparameters, architectures and solver choices).\n- Please also discuss sensitivity to your choices of hyperparameter choices, training hardware and times.\n- Please annotate the algorithms steps with comments so that it is easy to follow.\n- Bold the best numbers in Table 1 so that it is easy to parse.", "broader_impact_concerns": "I don't have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UGjEZGCvNN", "reviewer": "Reviewer_y3Mz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** \n - The paper effectively addresses a key issue in diffusion-based RL models:  computational inherent inefficiency during inference due to iterative reverse processes. By introducing Consistency Planning, the authors provide a valuable alternative that maintains high performance while drastically speeding up the inference process.\n - The integration of consistency models trained through distillation from pre-trained diffusion models into Decision Diffuser is an interesting concept. The authors clearly articulate the process of using inverse dynamics models to derive actions, showcasing how consistency models can be adapted from diffusion models for planning tasks.\n- The use of consistency models significantly reduces the number of steps required during inference, addressing the main challenge of slow sampling in diffusion models. This is a notable improvement, especially in real-time or resource-constrained environments.\n- The results on D4RL tasks provide evidence that the proposed method achieves comparable performance with diffusion-based methods while offering over a 12-fold speedup. The experiments compare to model-free approaches and to behaviour cloning and model-based methods, offering a broad evaluation framework. \n\n**Weaknesses:**\n - While the method section is detailed, it might be challenging for readers unfamiliar with diffusion and consistency models to follow. For example, the notation and equations could benefit from more intuition or explanation about how they contribute to the final results. The section on the inverse dynamics model and consistency distillation is mathematically dense, and a clearer breakdown of the concepts for general readers might improve accessibility.\n- While Consistency Planning offers a substantial speed improvement, its performance is slightly lower than state-of-the-art models like Diffusion-QL and Consistency-AC (87.9 vs. 85.1). The paper could provide more insights into why this trade-off occurs and under which conditions the slight drop in performance becomes significant.\n- The paper focuses on offline RL, but the application of consistency models in online RL settings could be a valuable extension and make the method more broadly applicable. The method's potential limitations or benefits in real-time decision-making tasks are not explored. The authors could discuss the potential challenges or benefits of applying the approach to scenarios where quick, real-time decision-making is critical.\n- The integration of multiple techniques (diffusion models, consistency models, inverse dynamics, etc.) might make the proposed method more challenging to implement in practice. More details on how these components interact and their impact on overall system complexity could be helpful for practitioners\n- While the performance results are positive, the paper primarily compares its method against diffusion-based baselines. More direct comparisons with other state-of-the-art model-based reinforcement learning approaches or those employing different planning strategies could further substantiate its claims. Additionally, while speedup is highlighted as a core advantage, a deeper dive into the computational costs (e.g., memory consumption, training times) compared to traditional diffusion models could add to the paper’s impact. Therefore I think providing a deeper analysis of the trade-offs between inference speed and performance would strengthen the paper. For example, investigating whether the slight drop in performance in some tasks is a general limitation of consistency models or specific to certain types of environments.\n- The future work outlined (e.g., designing better weighting functions, combining with advanced techniques in consistency models) could have been explored more. Even a brief experiment with one of these techniques could demonstrate the potential for further improvements, offering readers a better glimpse of what’s possible", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Consistency Planning, a novel approach that integrates consistency models with the Diffusion model framework to improve computational efficiency in model-based reinforcement learning (RL) for sequential decision-making tasks. Diffusion-based models, such as Diffuser and Decision Diffuser, have demonstrated strong performance but suffer from slow inference processes. The proposed method leverages the computational efficiency and speed of consistency models during the inference process, which significantly reduce the number of sampling steps required, without sacrificing the performance  and robust planning of the diffusion-based approaches. The method is validated on D4RL benchmark tasks, showing a 12-fold increase in speed compared to diffusion models, with comparable performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bFCT2QiTsL", "reviewer": "Reviewer_d7Jk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The introduction of consistency models enhances planning speed, which is vital for tasks with high real-time requirements.\n2. The authors have effectively demonstrated the efficacy of combining consistency models with the decision diffuser, although this outcome was somewhat predictable.\n\nWeaknesses:\n\n1. There appears to be a performance gap between the proposed method and Consistency-AC. It would be interesting to see how the method performs on long-horizon tasks, such as Maze2D.\n2. The time efficiency comparison with Consistency-AC is not clearly presented, which would help bolster confidence in the paper's claims.\n3. The paper lacks a discussion on potential implementation challenges. It's worth noting that open-source libraries like CleanDiffuser (https://github.com/CleanDiffuserTeam/CleanDiffuser) could potentially simplify the implementation of the proposed algorithm. However, this observation does not affect my overall decision on the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an approach that combines guided consistency models with the decision diffuser framework. Consistency Planning maintains performance comparable to diffusion model counterparts while significantly improving the computational efficiency of planning. This advancement is particularly crucial for tasks requiring agile motion planning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes that would impact recommendation for acceptance:\n\n1. Include experimental results for both the proposed method and comparison algorithms on long-horizon tasks, such as Maze2D.\n2. Submit the source code of the algorithm for review. This will allow for a thorough examination of the implementation details and facilitate reproducibility of the results.\n\nChanges that would strengthen the work:\n\n1. Elaborate on any challenges encountered during the implementation of the method, as well as potential issues that might arise when applying this approach to real-world robotic tasks.\n2. Provide a comprehensive time efficiency comparison between the proposed method and Consistency-AC.", "broader_impact_concerns": "I think there are no related ethical concerns about this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers recommended acceptance and I agree with their judgement.
Decision: Accept (poster)