Habitizing Diffusion Planning for Efficient and Effective Decision Making
Authors: Haofei Lu
Yifei Shen
Dongsheng Li
Junliang Xing
Dongqi Han
Venue: ICML 2025 poster (2025)
Keywords: Offline Reinforcement Learning
Diffusion Planning
Variational Bayes
Diffusion Models
Primary area: reinforcement_learning->batchoffline
TL;DR: A general framework for habitizing diffusion planning for efficient decision making

Abstract: Diffusion models have shown great promise in decision-making, also known as diffusion planning. However, the slow inference speeds limit their potential for broader real-world applications. 
Here, we introduce **Habi**, a general framework that transforms powerful but slow diffusion planning models into fast decision-making models, which mimics the cognitive process in the brain that costly goal-directed behavior gradually transitions to efficient habitual behavior with repetitive practice.   
Even using a laptop CPU, the habitized model can achieve an average **800+ Hz** decision-making frequency (faster than previous diffusion planners by orders of magnitude) on standard offline reinforcement learning benchmarks D4RL, while maintaining comparable or even higher performance compared to its corresponding diffusion planner. Our work proposes a fresh perspective of leveraging powerful diffusion models for real-world decision-making tasks. 
We also provide robust evaluations and analysis, offering insights from both biological and engineering perspectives for efficient and effective decision-making.

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Haofei Lu Yifei Shen? Dongsheng Li
Junliang Xing'
Dongqi Han 2
Abstract
Diffusion models have shown great promise in Socision-making also known as diffusion plan- ning. However, the slow inference speeds limit their potential for broader seal-world applications. Here, we introduce Habi, a general framework that transforms powerful but slow diffusion plan- ning models into fast decision-makee which mimics the cognitive process in the brain that costly goal-directed behavior gradually transi- tions to efficient habitual behavior with repetitive practice. Even using a laptop CPU, the habitized model can achieve an average 800+ Hz decision- making frequency (faster than previous diffusion planners by orders of magnitude) on standard of- fline reinforcement learning benchmarks D4RL, while maintaining comparable even higher per- formance compared to its corresponding diffusion planner. Our work proposes a fresh perspective of leveraging powerful diffusion models for real- world decision- recision-making tasks. We also provide robust evaluations and analysis, offering insights from both biological and engineering perspectives for efficient and effective decision-making.
1.0
Performance (Normalizad)
(Ours
0.9
DOL'
CL RL-R2*
0.7
models,
0.6
30
Adapiiiliuser
Decision Frequency (Hz)
Figure I. Performance Frequency. Performance is normal- ized across MuJoCo, AntMaze, and Kitchen tasks from D4RL. Decision frequency (Hz) measured on a laptop CPU (Apple M2, MacBook). Habitual Inference (HI), a lightweight model generated by our Habi, achieves an optimal balance between performance and speed. See Table I for more results.
fline reinforcement learning (RL) (Levine et al., 2020; Fu et al., 2020) by exploiting the generative power of diffusion models for making plans (trajectory of states). While recent diffusion model- based decision makers achieve state-of-the- art performance on standard offline RL benchmarks (Wang et al., 2023; Lu al., 2025), the computational cost of dif- fusion models remains a significant challenge models like diffuser and its variants usually take more than 0.1 seconds, sometimes even more than 1 second, to make a simple deci- sion using a decent GPU (Janner et al., 2021; Liang et al., 2023; Lu et al., 2025), which is unacceptable for real-world
1. Introduction
The trade-off between computational cost and effectiveness a key problem in decision making (Kahneman, 2011; Sidarus et al., 2019; Clark et al., 2021). In machine learning, probabilistic generative models have inereasingly been used for planning of the outcome of actions (Ha & Schmidhu- ber, 2018; Hafner et al., 2019; 2023). Recently, diffusion models have been used for decision making (Janner et al., 2021; Ajay et al., 2022; Du et al., 2024), in particular of-
applications.
Meanwhile, it has been widely known and researched that humans and animals can make optimal decisions using lim- ited energy. In cognitive science and psychology. decision making is often considered driven together by two kinds of behaviors (Dolan & Dayan, 2013; Wood & Rünger, 2016): a slow, deliberate goal-directed one, and a fast, automatic ha- bitual one. Goal-directed behavior, focuses on careful plan- ning and preeise evaluation of the future outcomes (the pro- also known as System 2 thinking (Kahneman, 2011)), making it more reliable but time-consuming. In contrast, the habitual behavior (System thinking (Kahneman, 2011)) se- lects actions in a model-fred manner - without considering subsequent outcomes, thus computationallly-ffiiiii while could be less reliable.
The work was conducted during the internship Haofei Lu (luhf23@@n shaails.singhua.edu.coo at Microsoft Research Asia 'Department of Computer Science and Technology. Tsinghua University 'Microsoft Research Asia. Correspondence to: Jun- liang Xing <jixing @wwe@ettisninuuuu..mm h microsofttooom..
Dongqi Han <dongqi-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
The habitized policy model is lightweighted and super-fast, providing orders-of-magnitude speedup over existing diffusion planners (Figure 1).
Efficiency:
Goal-directed Behavior
Habitual Behavior
(Beginner)
(Proficient)
Effectiveness: The habitized policy can compete with the state-of-the-art model that are much slower (Figure 1).
After Practice
Lookahead Planning
Direct Decisions
Versatility: Habi can be used straightforwardly for any diffusion planning and diffusion policy models.
We further conduct comprehensive evaluations across var- ious tasks, offering empirical insights into efficient and effective decision making.
X
Figure 2. An illustrative example of the process of habitization in playing the Minesweeper game. With practice, onc's 's decision- making relies less on deliberate goal-directed planning and more on context- dependent habitual behavior.
2. Bacckrround
2.1. Offline Reinforcement Learning.
In this study, we are inspired by one of the findings on habits and goals, known as habit formation, or habitization (Wood & Rünger, 2016; Han et al., 2024). That is, the bruin will gradually transform slow, deliberate goal-directed behavior into fast, habitual behavior when repetitively doing a task (see Figure 2 for an illustrative example).
Offline reinforcement learning (RL) (Fujimoto et al., 2019; Levine et al., 2020) considers the case that an agent learns from fixed dataset of previously collected trajectories with- out interacting with the environment. The objective is to train policy that maximizes the expected return, defined as Land Lh=07 "uloo"7a " perr++ when deployed in the environment. Here, rz is the immediate reward and y is the discount factor (Sutton & Barto, 1998). Offline RL is particularly relevant in scenarios where exploration is costly, risky, or im- practical, requiring agents to maximize the utility of existing data for careful planning. Key challenges include handling high-dime sigh-dimensional (Levine et al., 2020), long-horizon depen- dencies and deriving near-optimal policies from potentially sub-optimal datasets (Fujimoto et al., 2019). These factors position offline RI ideal benchmark for evaluating advanced decision-making methods. In this work, we evalu- ate our framework using standard offline RL benchmark D4RL (Fu et al., 2020), providing a rigorous and consis- tent comparison of its decision makion capabilities against previous baselines.
The key reason behind habitization is the hard need for a trade-off between efficiency (decision time and energy) and effectiveness (behavior performance) for animals to survive. A recent study (Han et al., 2024) proposes a com- putational framework to model the interaction between the two behaviors: As many real-world tasks are few-shot or zero-shot, animals need to utilize their world model to per- form goal-directed planning to make reliable decisions (Lee et al., 2014), which is computationally costly. To improve efficiency. they must meanwhile "extract" the goal-directed decision strategy to a habitual decision model that straight- forwardly makes decisions without planning by world mod- els.
We notice that diffusion planning (Janner et al., 2022; Ajay et al., 2022) is akin to goal-directed behavior, as both gener- ate plans in the future before making decision, and both are powerful but slow, hampering their real-world usage. Then, an idea naturally arise: Can we develop habitization pro- cess like in the brain to transform slow, effective diffusion planning into fast, foraigh habitual behavior?
2.2. Difffusion Models for Decision Making.
Diffusion models have recently demonstrated remarkable potential in decision-making tasks due to their ability to model complex distributions. Compared to classical diag- onal Gaussian policies (Haarnoja et al., 2018; Schulman et al., 2017; Kostrikov et al., 2021; Kumar et al., 2020), dif- fusion models have achieved ofate-of-the -rr performance in both online (Wang et al., 2024a; Yang et al., 2023; Psenka et al., 2023; Ren et al., 2024) and offline reinforcement learn- ing (Chen et al., 2023a: Li et al., 2023; Wang et al., 2023; Maansnn-Estruce et al., 2023; Janner et al., 2022). as well as demonstration learning (Ze et al., 2024; Chi et al., 2023). There are two main ways diffusion models are applied in decision-making:
In this work, we provide positive answer to this question. We develop a general framework, referred to as Habi, that mimics the brain's 's habitization process. Inspired by Han et al. (2024), Habi leverages variational Bayesian principles (Kingma Welling, 2014) to connect the slow, yet powerful diffusion planner's ' policy (as posterior) with a fast, habit- ual policy model (as prior). By maximizing the evidence lower bound (Kingma & Welling, 2014), the habitual policy model is trained to take advantage of pretrained diffusion planners while maintaining high efficiency, mimicking the habitization process in the brain (Wood& & Rünger, 2016).
(1) Diffusion planner (Ajay et al., 2022; Janner et al., 2022; Liang et al., 2023) models a trajectory T consisting of the
Habi featured with the following advantages:
2

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
current and subsequent H steps of states (or state-action pairs) in horizon:
terior distributions, respectively:
habitual action <- =
goal-directed action <- =
8t, $1+11+1 -54HH-1 Ct. 0r+1,""' Oz- 11
st, S1+1, -1+H-1 or
The intuition behind such formation is that posterior distri- bution in Bayes theory relies on additional information than the prior. Habitual behavior relies simply on context (current state), thus encoded as prior; whereas goal-directed behav- ior is refined by additional evidence (current state + plan of future states), thus encoded as posterior. The interplay between two behaviors (including the habitization process) can be modeled by minimizing a freee energy function (Fris- ton et al., 2006) (mathematically equal to the negative of ELBO (Kingma & Welling, 2014), also known as the deep variational information bottleneck (Alemi et al., 2017)):
(2) Difsiusion policy (Hansen-Estruch et al., 2023; Wang al., 2023) leverages diffusion models for direct action dis- tribution p(a,|s; (s)) modeling, and can be viewed as modeling trajectory [st au] with planning horizon H = 1.
Recent research has expanded the applications of diffusion planning to broader domains, such as vision-based decision- making (Chi et al., 2023) and integration with 3D visual rep- resentations (Ze et al., 2024). A recent study (Lu et al., 2025) provides comprehensive analysis of key design choices, of- fering practical tips and suggestions for effective diffusion planning. Our work, Habi, is orthogonal to diffusion plan- ners or diffusion policies, and can be viewed as an adaptive, general framework to habitize diffusion planning into effi- cient, habitual behaviors.
L BB)+)) Recoon. loss + RL loss]
DKL [+()))l((2],,
Accuracy
Complewity
where ((2) and p(z) denotes the posterior and prior proba- bilistic density function of z, respectively. Recon. and RL loss (accuracy) corresponds to learning the state-transition model and policy improvement, notably over the expec- tation using posterior 2. In the habitization process, the KI KL-divergence (complexity) term can be intuitively under- stood as aligning habitual behavior with the goal-directed one (A (Appendix B).
2.3. Auto-Encoding Variational Bayes
Variational Bayesian (VB) approaches in deep learning have been popular since the introduction of the variational auto- encoder (VAE) (Kingma& & Welling, 2014; Sohn et al., 2015). The core idea is to maximize the evidence lower bound (ELBO) of an objective function of probabilistic variable that we can replace the original distribution with vari- ational approximation based on a latent variable z (Alemi et al., 2017). The ELBO can be written as:
The Bayesian behavior framework was shown to replicate key experimental findings in cognitive neuroseience (Dolan & Dayan, 2013; Wood & Rünger, 2016), in an online RL setting. However, Han et al. (2024) used simple T-maze nav- igation tasks (O' Keefe & Dostrovsky, 1971; Olton, 1979) to compare with neuroscience experiments, and their methods have not been compared with state-of-the-art models from the machine learning community.
ELBO- = [e)) log P(x(z)) - DKL. [==)))l((z],, (1)
where p.q indicates the prior and posterior distributions of z, respectively. log P(x())) is the log-likelihood of cor- rectly reconstructing data x from 2, and DKL denotes Kull- back-Leibler (KL) divergence (Kullback & Leibler, 1951).
3.2. Habi: Framework to Habitize Diffusion Planners
Inspired by Han et al. (2024), we aim to solve the efficiency problems of diffusion planners by developing a framework, which we call Habi, to convert slow, careful diffusion plan- ning into fast, habitual actions, while keeping the effective- ness and the probabilistic nature of policy. Habi consists of two accordingly stages: (a) Habitization (Training) and (b) Habitual Inference, as depicted in Figure 3.
An important property of ELBO is that although the log- likelihood term in Equation| 1 is calculated with posterior samples ((=), the likelihood over prior distribution p(z) is also optimized with the KL-divergence term (see Ap- pendix B). Therefore, it is possible to reconstruct the data x using prior p(z) and the decoder <=)) after training.
Habi aligns the decision spaces of habitual behaviors and deliberate goal-directed (diffusion model-based planning, The habitual and goal-directed behaviors are encoded as the prior and posterior distributions of a latent probabilistic variable 2, respectively.
3. Methods
3.1. The Bayesian Behavior Framework
The Bayesian behavior framework (Han et al., 2024) pro- vides a mathematical model for the interaction between habitual and goal-directed behaviors. A latent Bayesian variable z encodes the two behaviors with its prior and pos-
As powerful diffusion planner models have already been de- veloped by existing studies (Lu et al., 2025), we can leverage the off-thh-sheeff, pretrained diffusion planning models.
3

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Pretrained Planner (Frozen)
St 1 Prior Encoder
Planner
Planning
Σr
MSE Loss
x N samples
St
2c
Seleet Best Using Critic
Prior Eicoddr
Posterior Ficodder
MSE Loss
z
Critic
-L-diiveeeecce
Loss
Decuder
Decoder
zt
at
(a) Habitization (Training)
(b) Habitual Inference
Figure 3. The diagram of Habi (a) During the Habitization (Training) stage, Habi learns to reconstruct actions from plans generated by a diffusion planner, with the decision spaces of habits (prior) and planning (posterior) aligned via KL. divergence in the latent space. Trainable parts include Prior Encoder, Posterior Encoder, Decoder, and Critic. (b) During the Habitual Inference (III) stage, only the lightweight prior encoder and latent decoder are required, enabling fast, high-quality habitual behaviors for decision-making.
3.3. Transition from Goal-directed to Habitual Behavior
where q[2|s, a²) is the posterior distribution representing goal-directed behaviors, and p(z(s) is the prior distribution representing habitual behaviors.
In this section, we will detail how behavior policy is habitized from given diffusion planner (Figure 3a). In our Bayesian behavior framework, the Bayesian latent variable 2f (or z9, where denotes prior, and q indicates posterior) is treated as a random variable following diagonal Gaus- sian distribution ((, of)) (or N Nu" )))). The subscript t denotes the timestep as we consider a Markov decision process (Bellman, 1957). For simplicity, we will omit the subscript for the following formulations.
The overall habitization loss is elegantly defined as (which is the famous (negative of) ELBO with adjusted KL weighting (Higgins et al., 2017)):
L Crecon + SKLCKL:
where 9KL a weighting factor balancing reconstruction ac- curacy and decision space alignment. fKI.. may vary greatly across tasks, and small 8KL. may lead to poor alignment, while large 9KI. may result in poor reconstruction (Hig- gins et al., 2017; Alemi et al., 2017). To enable habitization with minimal human intervention, Habi used an adaptive ÅKL mechanism (Haarnoja et al., 2018; Han et al., 2022) that dynamically adjusts value during training:
The prior distribution (/pp op) is obtained from mapping (feedforward network) from state s (Figure 3a):
zº ~ N (1e, <- PriorFncoder(ss)
(2)
The posterior 2 is trained to encode the goal-directed behavior by auto-encoding the diffusion planner's action a" (Figure 3a).
LaKI = log 3KL. -log1o LKI. - logio Dkrr)
(8)
28 ~ NN(H4,00) t- PosttrriorEncoder(s,. as)a")
(3)
where DÍe represents the target KL-divergence. This ap- proach bounds 3KL. to be in a reasonable range constrained by K leading to robust performance across tasks without requiring manual tuning (4.5).
A reconstruction loss Lrecon is introduced (We provide im- plementation details in Appendix Appendix C.3):
Lrecon Decoder(=")) a")] 1|22.
(4)
Intuitively, to make habitual behaviors consistent with goal- directed decisions, the decision spaces of the prior and poste- rior distributions are aligned under constraint by Kullback- Leibler (KL) divergence (Kullback & Leibler, 1951):
3.4. Supervising Habitual Behaviors with Critic
Habitual behaviors are fast and efficient but should not be purely instinct-driven. In the brain, regions like the dorsal striatum provide feedback to ensure they remain effective and avoid making mistakes (Kang et al., 2021). Inspired by (6) this, Habi also introduces a Critic function that evaluates habitual decisions by considering both the decision latent
LKI. DKL. [q(=|1, a" [][((]s)]]
= log Op + (a9)2 (110 12)2
σv 2(00)2
2'
4

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
and the corresponding action a (similar to Q-function, Figure 3a). The critic loss is defined as:
2022), AdaptDiffuser (Liang al., 2023), Decision Dif- fuser (DD) (Ajay et al., 2022), Diffusion Veteran (DV) (Lu et al., 2025), and (4) recent works on accelerated decision- making: DiffuserLite (Dong et al., 2024a), DTQL (Chen et al., 2024b). Considering that different baselines adopt varying frameworks and may differ in mereiiinnmmking frre- quency, we reproduce the relevant baselines (marked with *) using CleanDiffuser (Dong et al., 2024b). ensuring a fair and consistent comparison.
Laiitie = | - 2112
(9)
where Q is scalar, represents the decision quality. In of- fline reinforcement learning, Q is typically estimated from offline data and represent as a Q-function (Wang et al., 2023; Mlansen-Estruche et al., 2023), value function (Hansen- Estruch et al., 2023; Lu et al., 2025), or a classifier (Liang et al., 2023; Janner et al., 2022). In practice, we use the pre-trained Q corresponding to its diffusion planner as the ground truth of our critic function. Note that 29 is detached (gradient stopped) in critic learning.
Infrastructure All runtime measurements were conducted on two different computing hardwares: a laptop CPU (Apple M2 Max) or a server GPU (Nvidia A100). Training was on Nvidia A100 GPUs.
Reproducibility All the results are calculated over 500 episode seeds for each task to provide a reliable evaluation. HI''s results are additionally averaged on 5 training seeds to ensure robustness. Our code is anonymously available at attps::/bayesbbraat git io/.
3.5. Inference with Habitual Behaviors
After habitization training (Figure 3a), we obtained habit- ual behavior model, which can be used without planning (Figure 3b). We refer to it as Habitual Inference (Fig- ure 3b), which uses the Prior Encoder. Decoder, and Critic to generate habitual behaviors efficiently. HI samples multi- ple latents zf from the prior distribution, decodes them into corresponding actions ag, and select the best action using the critic. The process is formalized as:
4.2. Efficient and Effective Decision Making
How does HI compare with related methods in terms of efficiency and performance? By summarizing the results from the experiments (Table 1), we find that Habitual In- ference (HI), could achieve comparable performance of best diffusion policies or diffusion planners. Notably, HI even outperforms the strongest decision-making baselines in certain tasks, suggesting that the habitization process may potentially help mitigate planning errors and enhance decision quality. This phenomenon is particularly evident in tasks requiring precise planning, such as navigation and manipulation tasks.
{ fi=1 ~ NN o²) + PriorEncoder(s) (10)
0: = Decoder(z;)
(11)
a = arg max Critic(z,, a.). a;
(12)
where N represents the number of sampling candidates. The Critic evaluates each action ag along with its decision latent zi, and selects the best action the habitual behavior for inference. In practice, we observe using N = 5 candidates is sufficient to achieve satisfying performance. Detailed discussions and ablation studies are provided in Section 4.6.
Besides the standard performance metrics (the average total rewards in an online testing episode), real-world decision- making poses additional requirements on the decision effi- ciency or frequency. We also conduct a detailed analysis of the decision frequency across different tasks, hardwares, and parallelism levels in Table 4 in the Appendix 1. HI demonstrates a significant decision speed margin over other baselines. Figure 4 illustrates the trade-off between perfor- mance and frequency across different tasks, highlighting HI's ability balance efficiency and effectiveness.
4. Experimental Results
4.1. Experiment Setup
Benchmarks. We empirically evaluate Habi on diverse set of tasks from the D4RL dataset (Fu et al., 2020), one of the most widely used benchmarks for offline RL. We test our methods across different types of decision-making tasks, including locomotion, manipulation, and navigation (Appendix A).
4.3. Comparison with Direct Distillation Methods
How much performance gain can we obtain using Habi comparing with direct distillation of diffusion planner? We compare it with standard distillation methods. Specif- ically. we evaluate (1) HI w/o Critic, which removes the critic and directly uses the habitual policy without selection (Figure 3b), and (2) Standard Distillation, which applies supervised learning to mimic the planner. The results in Table show that even without the critic, Habi consistently
Baselines. To better compare the performance of Habi with other ofate-of-the-art diffusion planners to benchmark its performance, we include four types of representative baselines: (1) deterministic policies: BC (vanilla imitation learning), SRPO (Chen et al., 2024a) (2) diffusion poli- cies: IDQL (Hansen-Estiee et al., 2023). DQL (Wang et al., 2023), (3) diffusion planners: Diffuser (Janner et al.,
5

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Table Performance comparison on the D4RL benchmark. The reported values are Mean = Standard Error over 500 episode seeds for robust testing. Frequencies are measured using a CPU (Apple M2 Max. MacBook laptop). Baselines marked with * were reproduced using CleanDiffuser (Dong et al., 2024b) for consistency. Deterministic policies such as BC, SRPO (Chen et al., 2024a) are theoretically more efficient, however, their performance remains substantially inferior to probabilistic policies. The best results are highlighted in bold, while the second-best results are underlined. Action frequency on GPU can be found Table 4.
Tusks
Determinidic Policies
Pilfusion Policies
Diltusion Planners
Accelerated Probablissii Decision- Maaiing
Dataset
Eavironment
SRPO
IDQU
DQL
Difluser
Adqutiiliser
DD
DL-RI"
DE.R2
DTQL
(Dars)
Medicm Expert
HaltCheeeah
35.8
92.2
96.0000
2.3
89.400
905=
22.
90.6 1077
92.7 +2.2201.2
98.0
Malium-Repha
HallChaetah
38.4
51.4
0.3
000
07.7=0.5
38.1+00
4,33=43
4.18
4..
412
50.9=0.1
48.5=000
Median
60.4
52.3
0.3
4422
10.1
50,4
469
4:
57.
00.5
Medium- Fxpert
Hopper
111.9
101.1
+
1114402
1116+22.
11
1.0.000.
111:+002
0109+04
9.
Malium-Replay
Hopper
11.3
101.2
9.4
102:2+000
0.
922..15
100
91.9.00
0.
04.7
1000
102.0+00
Medium
Hopper
29.0
95.5
10.1
1023::00
14
9.64427
36
97.4
95.91110
996605
0025..0.
Mediam--appe Fapurt
Walkerdd
114
1106+00 0.0
111.+000
106.9
00
1.8
0100+00
1130
Medium Replay
11.8
84.6
29.
00
1.521.
857
4..
0,00
850
10200 0D
Medium
Walkerdd
66
64.4
1:.4
9..0
0.66
66.4226
10.5+14
8.
83
8.4
91.3
Performancy
41.
87.2
14.1
77.5
51.3
813
8333
14.1
K1.8
18.7
R02
Action Frequency
35.9
197.2
0.23
167
75.5
206.6
142.7
1329,7
Mixed
Ktichen
47.5
66.11.41
5 1.58
2.
518408
0.0
7.8
60
526110
00.20035
09
Partial
Kitchen
33.8
66.7 2.5
05.5
1.38
-3.721.3
55.5 2..
56.5
2.3
4..5
3784
24.2
94
Performance
00.3
56.1
53.7
65.8
838
53.0
45.2
67.3
K2.3
Action Frequency
CPU (Ha)
54.4
146.0
0,05
27.1
16.5
36.2
97.5
3N5.7
Diverse
Antmaze Large
0.0
53.6
40.0 1 11.4
3.7
27.3
25
0.000.00
90.0 1.8
0.0 000
326633
5600
25
Play
Lang
on
53.6
48.7 4.7
3.1
17.3 1.9
3.
0000000
2.
54.0 4.0
7134336
520022
$1.75177
Diverse
Astmaze Medium
75.0
83.3
3.0
2 1.6
60433
2.
87.4 1.6
2.8
2.
88.8
Play
Actmsso-Modiam
80.7
6.73+
6.7
150
8.0
80.0
7.3
780+
3.3
196=11
Performance
0.
65.7
99.8
5003
133
5.0
3D
85.2
54.7
67.1
67.0
WOL3
Action Frequency
CPU (Bix)
34.2
198.6
0.00
0.03
118
0.7
15.7
35.2
138.6
900,3
Lang
Mi
1.7
123
1670
2016
103.3 100.377.22
199.3
Medium
Ma
M03
1339
0520108
1215
42
150
52.14.6
1069
150. 1S
MonelD
1.8
110.6 4.1
140 1.0
113.9
1
116.6 1.5
36.4 7.0
12500669
144.1 17
Performance
13,0
140.3
199.8
1195
140.3
163.6
63.9
94.0
164.5
Action Frequency CPU (Ma)
215.8
1832.6
MuJoCo
Franka Kitchen
Antmaze
Mazeld
18/0
Porfomaato
Porformaac
Perferman
120
400
1000
120
100 80 Dedsion Frequency (H4)
136
430
43
006 Dedisoon Frequency (Htt)
-40
619
1009
Decidton Prequency (432)
Decition Frequency (14)
Figure 4. Visualized results Table 1. HI consistently performs in parallel with best models while being highly efficient.
outperforms standard distillation across all tasks.
We find that despite the fundamental differences in infer- ence mechanisms, the action distributions of HI remain well- aligned with those of DV across different states. The action distributions exhibit strong structural consistency, with HI capturing both the variability and intent of the planner's ac- tions. Notably, HI produces more concentrated distributions. This may reflect its ability to commit to high-confidence decisions without iterative refinement.
This demonstrates that HI itself contributes to effective decision- beyond simple imitation. While the critic further refines decision quality, the strong performance of HI w/o Critic confirms that Habi is not merely replicating planner behavior but instead leveraging the habitization pro- cess to learn an efficient and robust decision policy.
4.4. Visualizing Prior and Posterior Policy Distributions
4.5. Robustness of Adaptive KL- KL-Divergence Weighting
To investigate whether Habi can reasonably align habitual and goal-directed decision spaces we conduct case study by visualing the action distributions of the Habitual Infer- ence (HI) policy and its corresponding diffusion- plan- ner in the Maze2D environment, where the two-dimensional action space allows for a clear comparison. Figure 5 visual- izes the action distributions of Diffusion Veteran (DV) (Lu et al., 2025), a sfate-of-the art diffusion planner, and its cor- responding habitual inference policy generated using our framework Habi.
We used adaptive 9KL (Fquation 8) in habitization learn- ing for all tasks. Then a natural question is: how about manually tuning 9KT. on each task? We evaluate the ef- fectiveness of our proposed adaptive ŠKL. As shown in Figure 6, fixed 3KL tuning requires extensive grid search, and its performance is highly sensitive to the choice of (KL.- A suboptimal 9KI. can lead to either poor alignment (when too small) or degraded reconstruction quality (when too large). This issue is especially pronounced in complex tasks like AntMaze and Franka Kitchen, where improper
6

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Diffusion Planner
Habitual Inference (III) (IIl)
-05
-05
-14
-14
EEEEEE
E
State
t=0
75
150
225
300
Figure 5. Action distributions of Diffusion Planner (DV) and Habitual Inference (HI). Visualization of the action distributions from
a state-of-the-an diffusion planner (DV, 2.8Hz, top) (Lu et al., 2025) and its corresponding Habitual Inference policy (HI, 1532.6Hz, middle) generated by our Habi framework. Here, HII shows probabilistic generation capacity while roughly aligning with the action distribution from the diffusion planner (DV). More examples are deferred to Appendix D.4.
MuJaCo
Franka Kitchen
Antmaze
Maze2d
164
Tenomeess
Perfumaace
3.02
befa
KL DeDaa
K1. Ed.0
Floed X2-beta
w-w. Adsptive Ki-bera [0am
Fladd K--btta
www.
Acaptive Kibbeta iDung
Fboed KL-bebs
-ww. Adaettvve KL-zeta (Cung)
Fbond XL Kt-bets
Adlactive KLuzeta (Curc)
Figure Performance Comparison of Searching 3KL and our Adaptive 3KL Mechanism. Fixed (KI. requires grid search for each task to achieve peak performance, while our adaptive 9KL achieves comparable performance without task-specific tuning, shown by the consistent red dashed line across all tasks.
weighting significantly degrades performance.
shown in Figure 7, selecting from multiple candidates con- sistently improves decision quality across various tasks, confirming the benefit of leveraging the Critic for filter- ing. Interestingly, we observe that using a single candidate (N = 1) already achieves competitive results in some envi- ronments, demonstrating that Habi can function effectively even in an altra-lightweight setting. However, without can- didate selection, the policy loses the ability refine habitual decisions, leading suboptimal outcomes in more complex tasks like AntMaze and Maze2I). A moderate number of candidates (e.g., N = 5) provides good balance, achiev- ing near-optimal performance with minimal computational overhead.
In contrast, our adaptive 3KI. mechanism consistently achieves almost optimal performance across all tasks with- out requiring task-specific tuning. By dynamically adjusting 3KI. during training, it effectively balances decision space aligmment and reconstruction accuracy in an automated man- ner. This not only simplifies training but also ensures ro- bustness across diverse environments. The consistent per- formance of adaptive 3кI. (red dashed line) highlights its reliability in achieving strong results without extensive hy- perparameter tuning, making Habi potentially more scalable and adaptable across diverse tasks with minimal human intervention.
4.6. Importance of Action Selection with Sampling
5. Related work
We introduced action selection with sampling in HI (Fig- ure 3b), where more numbers of candidates will bring more computational cost. Here we examine the necessity of this design choice and the proper number of candidates. As
Accelerating Diffusion Decision Making Diffusion mod- els have been widely applied decision-making tasks, in- cluding offline RI, online RL, and imitation learning. How- ever, their slow inference speed remains significant bot-
7

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
MuJoCo
Franka Kitchen
Table Comparison with alternative approaches. We compare HI with (I) HI without citic-based selection; and (2) standard distillation, which directly applies supervised learning to distill the corresponding diffusion planner.
Meorgo Porfomonne
37.0
812
86.0
885
87.
67.5
Averapp Paaformnac
825
122
0177
916
Tasks
Methods HI w/o Critic
Dataset
Environment
HT (Ours)
Direct Distill
Med-Exp
HaliCheetah
98.0 0.0
2.1
97.1 a 0.1
100
Candiddles
100
Med. Rep
HaliCheetah
48.5 0.0
0.0
46.7
Medium
HalfCheetah
0.0
51.2+
51,2 0.1
Antmaze
170
Maze2d
Med-Exp
Hopper
92.4 2.0
8.0
61.1 = 1.5
10.3
797
Pufarmancs
Med Rep
Hopper
0.0.0
1
0,2
Medium
Hopper
0.2.5+0.1
98.5 1.1
102.2 a 0.0
Med-Exp
Walker2d
0.3.0
0.0.5
112.2 0.0
Ansnag
Med-Rop
Walker2d
000
0.2
0000
150
Medium
Walker2il
9.1
91,4
9..9
Performance
89.2
(-2.0%)
(-4.9%)
10
20
50
102
100
10
20
100
Mixed
Kitchen
0.4
6.46
69.1 = 0.4
Num Chand
Num Carciibaice
Partial
Kitchen
94,8
9..2
64.8 = 0.8
Performance
82.3
67.01-19.115 ((2.3%)
Figure 7. Effect of the number of sampling candidates per- formance. Increasing the number of candidates N improves per- formance initially, but excessive candidates (NV 2 50) lead to diminishing returns or slight degradation. A moderate choice (e.g.. N 5) provides good balance between performance and effi- ciency. Note that N = is the case without critic-hased selection.
Diverse
Antunazo-Large
2.0
0.8
72.0 1.6
Play
Antmaze-Large
81.7 1.7
77.8 1.8
$2.0 1.6
Diverse
Antmaze-Medium
88.8 2 1.4
9.1 1,1
44.0 1.5
Play
Antmaze-Medium
85.3 1.5
88.0 1.4
76.0 1.2
Performance
80.3
65.4 (-18.5%)
(-29.44))
Large
Maze20
2 199.2
19
Medium
M
1
143.9 1.6
143,0 1.7
Umaze
Maze2D
1 1.7
138.3 1.8
137,7 # 1.8
RL objective functions (Levine, 2018; Fellows et al., 2019; Guan et al., 2024). Other usages include using the predic- tion error to compute curiosity-driven reward to encourage exploration (Yin et al., 2021). Moreover, Han et al. (2022: 2024) demonstrated that ELBO can connect two policies in online RL, with the one easier to learn as posterior, and the prior could be used for inference, which inspires our study, However, Habi focuses on offline RL, and we are the first to show such an ELBO can result in tremendous speedup without significant performance degradation.
Performance
164.5
161.3 (-1.9%)
159.9 (-2.9%)
tleneck, limiting their broader applicability in real-world scenarios. A primary approach to mitigating this issue in- volves specialized diffusion solvers (Song et al., 2020; Lu et al., 2022) to reduce sampling steps, while varying degrees of performance degradation remain unavoidable. More re- cently, distillation-based approaches (Poole et al., 2023; Wang et al., 2024b) have been proposed to enable one-step generation in diffusion models. However, directly applying these techniques to decision- decision-making may lead to suboptimal performance (Chen et al., 2024b:a).
6. Conclusion
To achieve both effective and efficient decision-making, ODE reflow-based numerical distillation has been intro- duced as a promising direction (Dong et al., 2024a). Mean- while, recent studies also explore joint training of diffusion model and a single-step policy, using the latter for deploy- ment while maintaining state-of-thed of-t performance (Chen et al., 2024b). These works serve as important baselines for our comparison.
In this work, we introduce Habi, general framework that habitizes diffusion planning, transforming slow, goal- directed seecision-making processes into efficient, habitual behaviors. Through comprehensive evaluations on standard offline RL benchmarks, we demonstrate that Habi achieves orders-of -ffrs-oofmmn mf-naniiuude speedup while maintaining or even sur- passing the performance of ofate-of-the-art diffusion-based decision-making approaches.
Looking forward, Habi demonstrates potential for real-time, high-speed decision-makinn in real-world environments, making it a promising approach for applications in em- bodied AI and other real-world decision-making problems. Nevertheless, our work has limitations. We have focused on state (proprioception)-beee bassd decision-making tasks using offline datasets. Future directions include extending Habi to online RI., investigating its generalization to broader domains such as vision-based tasks (Du et al., 2024; Chi et al., 2023), examining its effectiveness in worl-world tasks, as well as combining with orthogonal techniques such as ensemble approaches (An et al., 2021; Yang et al., 2022).
Variational Bayes in RL Variational Bayesian (VB) meth- ods have been incorporated into RL. research for several pur- poses. A popular usage is exploiting the generative capacity of VAE and its recurrent versions (Chung et al., 2015) for constructing the world model (Ha & Schmidhuber, 2018: Hafner et al., 2018; 2023). VB methods are also used to extract useful representation of environmental states from raw observations by modeling the state transitions (Igl et al., 2018; Han et al., 2020; Ni et al., 2024). The acquired rep- resentation is then used for the original RL task, i.c max- imizing rewards. Also, VB principles can help to design
8

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Impact Statement
Clark, L., Robbins, T. W., Ersche, K. D., and Sahakian, B. J. The computational cost of active information sam- pling before decision-making under uncertainty. Nature Human Behaviour, 5(6):789 800, 2021. doi: 10.1038/ s4 s41562-021-01116 16-6.
This work contributes to technical advancements in ma- chine learning by proposing an efficient decision-making framework. It does not introduce unique ethical concerns beyond standard considerations in algorithmic development and deployment.
Dolan, R. J. and Dayan, P. Goals and habits in the brain. Neuron, 80(2):312-33 2013.
References
Dong, Z., HAO, J., Yuan, Y., Ni, F., Wang, Y., Li, P., and ZHENG, Y. Diffuserlite: Towards real-time dif- fusion planning. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024a. ht.pts::: http///pppeeeeeeeeeeeeeeeeeeeeet net/ forum? Ad-2TXDHUGyrQ.
Ajay, A., Du, Y., Gupta, A., Tenenbaum, J. B., Jaakkola, T. S., and Agrawal, P. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2022.
Dong, Z., Yuan, Y., HAO, J., Ni, F, Ma, Y., Li, P., and ZHENG, Y. Cleandiffuser: An easy-to-use modular- ized library for diffusion models decision making, The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024b. URL wwtph:ttt///// htt.ttts:///ppeeeeeeeee /oppnreveeeeeeeeeeeeeeeeeereeewe net/ forum? id id-7ey2ugXs36.
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. In International Conference on Leurning Representations, 2017.
An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty- based offline reinforcement learning with diversified q- ensemble. Advances in neural information processing systems, 34:7436-7447 2021.
Du, Y.. Yang. S., Dai, B.. Dai, H., Nachum, O., Tenenbaum, J., Schuurmans, D., and Abbeel, P. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
Bellman, R. A Markovian decision process. Journal of Mathematics and Mechanics, pp. 679-684, 1957.
Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline rein- forcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, 2023a.
Fellows, M., Mahajan, A., Rudner, T. G., and Whiteson, S. Virel: A variational inference framework for rein- forcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Chen, H., Lu, C., Wang, Z., Su, H., and Zhu, Score regu- larized policy optimization through diffusion behavior. In The Twelfth International Conference on Learning Rep- resentations, 2024a. URL https: /pppneevee...e.e fort/form? d=xCRr x 9DrolJ.
Friston, K., Kilner, J., and Harrison, L. A free energy principle for the brain. Journal of Physiology-Paris, 100 (1-3):70-87 70, 2006.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Chen, T., Wang, Z., and Zhou, M. Diffusion policies creat- ing a trust region for offline reinforcement learning. arXiv preprint ar Ar05 19690, 2024b.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforeement learning without exploration. In Interna- tional Conference Machine Learning, pp. 2052-2062. PMLR, 2019.
Chen, Y., Li. H., and Zhao, D. Boosting continu- ous control with consistency policy. arXiv preprint orXiv:2310.06344 2023b.
Guan, J., Chen, G., Ji, J., Yang, L., Li, Z., al. Voce: Variational optimization with conservative estimation for offline safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song. S. Diffusion policy: Visuomo- policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241123366. 2023.
Ha, D. and Schmidhuber, J. Recurrent world models fa- cilitate policy evolution. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Gar- nett, R. (eds.), Advances in Neural Information Process- ing Systems 31, pp. 2450-2462. Curran Associates, Inc., 2018.
Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C.,, and Bengio, Y. recurrent latent variable model for sequential data. In Advances in Neural Information Pro- cessing Systems, pp. 2980 2988, 2015.

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.088
Kang, J., Kim, H., Hwang, S. H., Han, M., Lee, S.-H., and Kim, H. F. Primate ventral striatum maintains neural rep- resentations of the value of previously rewarded objects for habitual seeking. Nature communications, 12(1):2100, 2021.
2018.
Hafner, D., Lillicrap. T., Fischer, I., Villegas, R., Ha, D.,
Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 1.00555, 2018.
Kingma, D. P. and Welling, M. Auto-cncoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019.
Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I,, and Welling, M. Improved variational infer- ence with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXiv:2301. 104,, 2023.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint 1rXiv:2110.061666 2021.
Han, D., Doya, K., and Tani, J. Variational recurrent models for solving partially observable control tasks, In Interna- tional Conference on Learning Representations, 2020.
Kullback, S. and Leibler, R. A. On information and suf- ficiency. The Annals of Mathematical Statistics, 22(1): 79-86, 1951.
Han, D., Kozuno, T., Luo, X., Chen, Z.-Y., Doya, K., Yang. Y., and Li, D. Variational oracle guiding for reinforce- ment learning. In International Conference on Learning Representations, 2022.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1199-1191, 2020.
Han, D., Doya, K., Li, D., and Tani, Synergizing habits and goals with variational Bayes. Nature Communica- tions, 15(1):4461, 2024.
Lee, W., Shimojo, S., and O' D'Doherty, J. P. Neural compu- tations underlying arbitration between model-based and model-free learning. Neuron, 81(3):687-699 2014.
Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor- critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Levine, S. Reinforcement learning and control as proba- bilistic inference: Tutorial and review, 2018.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166: 2020.
Higgins, I. Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (-VAE: Learning basic visual concepts with a constrained varia- tional framework. In International conferecce on learning representations, 2017.
Li, W., Wang, X., Jin, B., and Zha, H. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pp. 20035-20064. PMLR, 2023.
Igl, M., Zintgraf. L.. Le, T. A., Wood, F., and Whiteson, S. Deep variational reinforcement learning for POMDPs. arXiv preprint arXiv:1806 (tt.:.820002266, 2018.
Liang, Z.,, Mu, Y., Ding, M., Ni, F,, Tomizuka, M., and Luo, P. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. ICML, 2023.
Janner, M., Li, Q., and Levine, S. Offline reinforcement learning one big sequence modeling problem. In Ad- vances in Neural Information Processing Systems, vol- ume 34, 2021.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances Neural Information Processing Systems, 35:5775-5788 2022.
Janner, M., Du, Y., Tenenbaum, J., and Levine, S. Planning with diffusion for flexible behavior synthesis. In Interna- lional Conference on Machine Learning, pp. 9902-9915. PMLR, 2022.
Lu, H., Han, D., Shen, Y., and Li, D. What makes good diffusion planner for decision making? In The Thirteenth International Conference on Learning Representations, 2025. URL https: teetts::/oppnzeeeee /openrevenwweeeeeeeeeeeeeeeeeeee net forum? i id=78gkxxM8Fy.
Kahneman, Thinking, fast and slow. macmillan, 2011.
10

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Ni, T., Eysenbach, B., SeyedSalehi, E., Ma, M., Gehring, C., Mahajan, A., and Bacon, P.-... Bridging state and history representations: Understanding self-predictive RL. In The Twelfth International Conference on Learning Representations, 2024. URL. https: /pppnrevie..eee.ee..eeeee....... net f forum? ma
Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., and Zhu, J. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024b.
Wood, W. and Rünger, D. Psychology of habit. Annual review of psychology, 67:289-314, 2016.
O' J'Keefe, J. and Dostrovsky, map: preliminary evidence from unit activity in the freely- moving rat. Brain research, 1971.
J. The hippocampus spatial
Yang, L., Huang. Z., Lei, F., Zhong, Y., Yang, Y., Fang. C.,
Wen, S., Zhou, B., and Lin, Z.. Policy representation via diffusion probability model for reinforcement learning, arXiv preprint arXiv:2305.. 20:::2335.11222 13122, 2023.
Olton, D. Mazes, maps, and memory. American psychol- ogist, 34(7):583, 1979.
Yang, R., Bai, C., Ma, X., Wang, Z., Zhang, C., and Han, Rorl: Robust offline reinforcement learning via con- servative smoothing. Advances in neural information processing systems, 35:23851 2022.
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dream- fusion: Text-to-3d using 2d diffusion. The Eleventh International Conference on Learning Representations, 2023. URL /ttp://////eeeeeeeeeeeeee net/ forum?
Yin, H., Chen, J., Pan, S. J., and Tschiatschek,, S. Sequen- tial generative exploration model for partially observable reinforcement learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pp. 10700- 10708, 2021.
Psenka, M., Escontrela, A., Abbeel, P., and Ma, Y. Learn- ing diffusion model policy from rewards via Q-score matching. arXiv preprint 1r11v 2023.
Ren, A. Z., Lidard, J., Ankile, L. L., Simeonov, A., Agrawal, P., Majumdar, A., Burchfiel, B., Dai, H., and Simchowitz, M. Diffusion policy policy optimization. arXiv preprint 0rXiv:2409.00588, 2024.
Ze, Y., Zhang, G., Zhang, K., Hu, C., Wang, M., and Xu, H. 3D diffusion policy, arXiv preprint arXiv:2403.033: 033554444. 2024.
Zhang, Z., Sun, Y., Ye, J., Liu, T.-.., Zhang, J., and Yu, Y. Flow to better: Offline preference based reinforce- ment learning via preferred trajectory generation, The Twelfth International Conference on Learning Represen- lations, 2024. URL https: openreview net/ forum? 1d-GG68SZ2LT.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.063447 2017.
Sidarus, N., Palminteri, S., and Chambon, V. Cost-benefit trade-offs decision-makee and learning. PLoS Com- putational Biology, (5(9))e100733))) 2019. doi: 10.1371/ journal.pchi. 1007326.
Sohn, K., Lee, H., and Yan, X. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Sutton, R. and Barto, A. G. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Wang, Y., Wang, L., Jiang, Y., Zou, W., Liu, T., Song, X., Wang, W., Xiao, L., Wu, J., Duan, J., et al. Diffu- sion actor-critic with entropy regulator. arXiv preprint arXiv:2405.15177 2024a.
Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
A. Benchmarking Tasks
As shown in Figure 8, we consider a diverse set of benchmarking tasks to evaluate the performance of Habi. These tasks are designed to encompass wide range of evaluation metrics, including locomotion tasks that emphasize short-term planning. robotic arm tasks requiring long-term strategic planning, and navigation tasks focused on path planning, We provide detailed description of each task below.
MuJoCo Walker2D
MuJoCo HalfCheetah
MuJoCo Hopper
F G
Franka Kitchen
Antmaze
Maze2D
Figure 8. Rendering of the benchmarking tasks considered in this study. The tasks encompass a diverse set of evaluation metrics, including locomotion tasks that emphasize short-term planning, robotic tasks requiring long-term strategic planning, and navigation tasks focused on path planning.
MuJoCo Locomotion The MuJoCo Locomotion task standard benchmarking task in reinforcement learning that requires the agent to control simulated robot to navigate through complex environment. the agent's ability to perform locomotion tasks that require short-term planning.
The task is designed to evaluate
Franka Kitchen The Kitchen task is a robotic arm manipulation task that requires the agent to manipulate objects in a kitchen environment. The task is designed to evaluate the agent's ability to perform long-term strategic planning and manipulation tasks.
AntMaze The AntMaze task is combination of locomotion and planning tasks that require the agent to navigate through maze environment. The variations of this environment be initialized with different maze configurations and increasing levels of complexity. The task is designed to evaluate the agent's ability to perform locomotion tasks while incorporating planning strategies.
Maze2D The Maze2D task a pure navigation task that requires the agent to navigate through a 2D maze environment. The variations of this environment can be initialized with different maze configurations and increasing levels of complexity. These tasks are used to test planning capabilities in environments where spatial reasoning critical.
B. Property of ELBO
While the ELBO (Equation 1) optimizes the log ligeelidl of reconstruction data, Ez-y(z) log P(x(2)), based on the exception over posterior distribution of the likelihood over the expectation of prior distribution, Ez-p(z) P(x(2)). is also being optimized. Equivalently, we need to show that the ELBO is indeed a lower bound on log Ex~)(s) P(x|2):
E Ex>g(=) log P(x|z)- - DKL [a(=)][[(=)] < log Ezmp(e) P(a|z).
(13)
We start with the definition the marginal likelihood:
log Eepp(a) P(x) = log / pl =)) log ((=))))(=)dz.
(14)
Introducing a variational distribution q(2), rewrite this using importance sampling:
log Bs-p(t) P(x) log / ((<)) p(x(2)p(2) dz.
(15)
<(2)
12

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Applying Jensen's inequality (since log is concave), we obtain:
log E (()p 2 (() p(zl2))p(2)
g(2) dz.
(16)
Rewriting the eight-hand side:
(s)) logp(z|>) + (x) log p(z)
(17)
g(2)
Recognizing the second term as the negative Kullback-Leibler (KL) divergence:
(x)(()) logp(z|2) Dka(
(18)
Since the KL divergence is always non-negative, conclude that:
Bzwg(=) (((l))) DkL(I())lp(z)) s log (xmp(=))(ar))
(19)
Thus, the ELBO a valid lower bound on log P(x)). This ensures that our ELBO-like objective function (Equation 7) improves habitual (prior) policy.
C. Implementation Details
In this section, we provide additional implementation details of Habi, including the network architecture and hyperparameters used in all experiments.
C.1. Network Architecture
As demonstrated in Figure 3, the Habi pipeline consists of four main components: (1) Prior Encoder, (2) Posterior Encoder, (3) Decoder, and (4) Critic. All these modules can be implemented with a lightweight multiple-layer perceptrons (MLP) and trained using standard backpropagation during the Habitization stage. We provide detailed descriptions each component below.
Prior Encoder The Prior Encoder is required encode the state into the prior latent 2 for generating habitual behaviors. In Habi, it is implemented with multiple-layer perceptron (MLP), that only takes the current state as input and outputs the mean yd and standard deviation oº of the prior latent z2 (NP,, a)):
Nº MLP((s)
_" (T.
(20)
o" softplus(E") +e,
where is small constant to ensure numerical stability (c 0.01 in our work).
Posterior Encoder The Posterior Encoder is responsible for encoding the state-action pair (s, a) into the posterior latent 29 for generating goal-directed behaviors. is implemented similarly to the Prior Encoder, with the only difference being that it takes the ttate-action pair input:
4 MLP(se, a*)
MLP(s a*)
(21)
σ softplus({9) +
Decoder The Decoder responsible for decoding the decision latent into action In Habi. the Decoder is implemented simple MLP that takes the latent input and outputs the action
MLP(z).
(22)
13

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Critic The Critic is used to evaluate the quality of habitual behaviors generated by the Decoder. In Habi, the Critic is implemented as simple MIP that takes the decision latent and the action input and outputs scalar value Q representing the decision quality:
2 MLP(2,
(23)
In all experiments of this work, these components are consistently implemneted as lightweight three-layer MLP with hidden dimension of 256, as is shown in Table 3. The Direct Distill baseline uses the same architecture Habi but performs imitation learning directly state-action pairs provided by diffusion planners.
C.2. Full Hyperparameters
We provide the consistent hyperparameters used in all experiments in Table 3.
Table 3. Hyperparameters in experiments.
Settings
Value
Optimizer
Adam
Learning Rate
3e-4
Gradient Steps
1000000
Batch Size
256
Latent Dimension: Dim(z)
256
MIP Hidden Size (Encoder & Decoder)
256
MLP Hidden Layers (Encoder & Decoder)
2
Habitization Target (Locomotion Related)
DQL (Wang et al., 2023) (MuJoCo, Antmaze)
Habitization Target (Planning Related)
DV (Lu et al., 2025) (Kitchen, Maze2D)
Target KL--iiveeeecce Dir.
1.0
Number of Sampling Candidates in Habitization training
50
Number of Sampling Candidates in Habitual Inference 5
C.3. Policy and Critic Training Details
To provide further details on the loss functions introduced in Section 3.3 and Section 3.4, we elaborate on the reconstruction loss (Equation (4)) and the critic training loss (Equation (9)) in this section.
Policy Training. Most state-of-the-art diffusion model-based policy models generate actions by sampling a batch of candidate actions and selecting the best one for decision scarchin.. recent studies (Wang et al., 2023; Dong et al., 2024b; Lu et al., 2025) also have validated its effectiveness. In Habi, we aim to learn a policy that directly habitize their final decisions. Specifically, the reconstruction loss optimizes the decoder to recover the planner's 's selected action:
Crecon Decoder(=") - a" ||2.
(24)
where a" is the best action chosen by the planner from the candidate batch. This helps the habitual policy focus on habitizing high-quality decision-making rather than reproducing the full distribution of candidate actions, ensuring efficient and reliable behavior generation.
Critic Training. Unlike the policy, which only needs to learn the planner's final decision, the critic benefits from exposure both optimal and suboptimal plans. A robust critic should distinguish between high- and low-quality decisions, enabling more effective filtering of habitual actions during inference. To achieve this, we train the critic using the whole batch of candidate actions. Formally, the critic loss is given by:
Leritic
Critic(=",aa] 2.1l2
(25)
14

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
where aj also includes suboptimal actions that selected by the planner, and Q, is the corresponding value given by the pretrained value function of the diffusion planners (Wang al., 2023; Lu et al., 2025). This exposure diverse plans allows the critic to generalize better and maintain robustness in habitual decision-making.
D. Extensive Experimental Results
D.I. Decision Frequency Evaluation across Different Devices
We evaluate the decision frequency of our methods across tasks, devices, and levels of parallelism (Table 4). On a PC laptop (Apple M2 Max CPU), HI consistently achieves the highest decision frequency, significantly outperforming diffusion policies (IQQL,, DQL.) and diffusion planners (Diffuser, AdaptDiffuser, DD, DV). Compared with other accelerated probabílistic decison-making methods, HI demonstrates a consistent advantage in decision frequency across all tasks.
On a server with an Nvidia A100 GPU, HI maintains its advantage of high decision frequency. At 10 and 20 parallel environments, it leads Franka Kitchen, Antmaze, and Maze2I). In MuloCo with 10 environments, DTQL achieves a slightly higher frequency, but HI remains competitive while being significantly faster than diffusion- methods.
The results demonstrate that HI is capable of making decisions at high frequency across different tasks, devices, and levels of parallelism, making it suitable choice for real-time decision- decision-making applications.
Table Decision-making frequency across tasks, devices and parallelism levels. Frequency reflects the number of actions (or action batches) generated per second using either PC laptop (CPU@ Apple M2 Max) or a Linux server (GPU@ Nvidia A100) under varying levels of parallelism. The best results are highlighted in bold.
Task
Device
Parallel Eays
IDQL
DQL*
Diffuser
AdaptDiffuser
DD
DV
DL-RI*
DL-R2*
DTQL
HI (Ours)
PC (CPU)
35,9
197.2
0.2
0.2
16.7
7.5
75.5
206.6
142.7
1329.7
MuloCo
Server (GPU)
10
110.9
135.4
3.1
22.9
18.0
49,4
109.2
318.8
308.4
Server (GPU)
20
86.5
108.1
3.0
3.0
22.4
19.8
48.4
104.2
223.7
281.3
(CPU)
34.4
146.0
0.1
0.1
27.1
2.7
16.5
36.2
97.5
385.7
Franka Kitchen
Server (GPU)
10
78.4
92.9
2.3
2.4
22.1
7.8
27.2
51.0
143.6
147.3
Server (GPU)
20
64.2
85.2
2.2
2.2
21.5
4.2
24.0
36.3
113.0
137.3
PC (CPU)
1
34.2
198.6
0.03
0.03
11.8
0.7
15.7
35.2
138.6
908.3
Antmaze
Server (GPU)
10
102.3
127.6
2.5
2,3
24.3
2.0
29.8
58.0
241.4
261.4
Server (GPU)
20
80.1
100.1
1.5
1.5
24.2
1.0
24.9
50.8
196.3
251.5
PC (CPU)
33.9
215.8
0.03
0.03
2.8
16.4
38.5
1532.6
Maze2D
Server (GPU)
10
126.4
151.9
2.7
2.6
8.3
30.3
62.5
388.4
Server (GPU)
20
90.8
140.8
1.5
1.5
4.4
24.8
52.6
376.0
D.2. Impact of Number of Candidates
To further analyze the role of candidate selection, we provide additional results in Table 6, extending our discussion from the main text. The results confirm that selecting from multiple candidates consistently improves decision quality across various tasks. Even with single candidate (N 1), HI remains competitive, reinforcing its efficiency in lightweight settings. However, increasing N enables further refinement, particularly benefiting complex tasks such as AntMaze and Maze2D, where decision quality is more sensitive to suboptimal habitual actions.
We also observe diminishing returns beyond moderate number of candidates. While larger N improves performance, gains plateau after 5 in most environments, suggesting that a small candidate pool is sufficient for near-optimal decision-malee This balance allows HI to achieve high efficiency without excessive computational overhead, making it adaptable across different settings.
D.3. Additional comparison results
We also provide additional comparisons with accelerated generative decision Docision-making methods, including flow-based (Zhang et al., 2024; Dong et al., 2024a), consistency-bases methods (Chen et al., 2023b) on shared tasks according the original papers. As shown in Table 5, our method (HI) achieves consistently strong performance across a range of MuJoCo environments. These results highlight the effectiveness of our approach in balancing planning quality and efficiency across diverse offline settings. However, do not want to convey the message that Habi is better than blowd method. Indeed, they are orthogonal and can be combined (Kingma et al., 2016) for future work.
15

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Table Additional comparison results tasks of MuJoCo
Environment
HI (Ours)
FTB
CPIQL
CPQL
HalfCheetah-ME
0.0
85.2 + 0.7
81.0 = 1.7
9.5
HalfCheetah-MR
0.0
38.4 1.3
1.4
46.6
HalfCheetah-M
0.0
1.0
56,9
Hopper-ME
111.1 2.0
1.0.6
110.4
Hopper-MR
1.0
4.9
1.0
97.7
Hopper-M
0.1
2.0
4.5
Walker-ME
113.0
109.3 0.3
110.9 0.2
Walker-MR
102.0 0.0
79.1 + 1.4
91.8 + 2.8
9366
Walker-M
91.3 # 0.1
86.2 # 0.6
82.1 + 2.4
Tasks Average
89.2
87.0
88,4
Shared Tasks Average
92.7
85.5
90.5
92.8
Table Sweeping Number of Sampling Candidates. This table presents the impact of critic selection and varying the number of candidates N on performance across different datasets and environments. The reported values Mcan Standard Error over 5 training seeds x 500 episode seeds.
Tasks
Number of Candidates
Dataset
Environment
=
N =
N =
N
=
N 50
N 100
Medium- Expert
HalfCheetah
9.1
0.1
9.00
0.0
80.0
98.6
98.7
Medium- Medium-Replay
HalfCheetah
4..0
47.9 0.0
4.0
0.0
4.0
48.9
49.0 0.0
Medium
HalfCheetah
5..2+0.0
5.0
0.0
0.0
54.8 0.00
0.04
55.8
Medium Medium-Expert
Hopper
8.09
92.7
9.0
8.06
83.1
77.7 1.9
74.0 1.7
Medium Replly
Hopper
0.0
101.9
102.0
0.0.0
102.0 1.0 0.0
Medium
Hopper
98.5 1.1
100.2
0.1
0.2.7
102.9
102.6 0.2
102.7 = 0.2
Medium-Expert
Walker2d
112.5 0.0
112.7 0.0
0.0
113.1
0.0
113.4 0.0
113.5 0.0
Medium Replay
Walker2d
101.9 a 0.2
102.0 0.0
102.0&100
101,9 0.0
101.7 a 0.0
101.5 + 0.0
101.5 & 0.0
Medium
Walker2d
91.4 + 0.3
91.8 a 0.2
91.3 0.1
91.1
0.1
90.6
90.3 0.4
Performance
89.0
88.5
87.9
87.5
Mixed
Kitchen
66.6 t 0.4
68.4 # 0.4
69.8 & 0.4
70.4 0.4
70.6 a 0.3
70.2 & 0.4
70.3 0.4
Partial
Kitchen
a
94.8 0.6
94.2
93.8 0.6
93.2
Performance
80.4
82.0
82.3
82.3
82.2
81.7
81.6
Diverse
Antmaze-Large
3.8 0.8
42.3
65.2 2.0
70.0 2.0
72.9 1.9
71.7 2.0
71.3 2.0
Play
Antmaze-Lare
77.8 +
82.4 1.7
81.7 1.7
81.0 1.7
80.4 1.7
77.8 1.8
73.4 1.9
Diverse
Antmaze- Antmaze-Medium
92.1 + 1.1
92.4 1.1
88.8 1.4
83.8 + 1.6
78.8 1.8
7.00
64.7 2.1
Play
Antmaze Mediuu
88.0
1.5
84.0 1.6
84.2 = 1.6
84.6 1.6
$2.3 1.7
Performance
65.4
76.1
80.3
79.7
79.1
76,3
72.9
Large
Maze2D
201.8 1.9
203.5
1.8
199.2
2.0
194.3 2.1
193.2 2.1
187.7
183.9 2.4
Medium
Maze2D
143,9 + 1.6
149.1
1.5
150.1
1.5
149.8 1.5
148.4 1.6
149.0 0 1.6
146.4 1.7
Umaze
Maze2D)
138.3 1.8
141.5 1.7
144.3 1.7
146.2 1.7
143.5 1.7
144.5 1.6
145.0 1.7
Performance
161.3
164.7
164.5
163.4
161.7
160.4
158.4
16

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
D.4. Visualizations of Habitual and Goal-Direen Coal-Direccted Policy Distributions
Diflission Planner
Habitual Inference (11I)
Md am sa9
103
100
0.0
330
030
x25
375
0.5
900
109
State
t= 0
t= 25
t= 50
t= 75
t= 100
t= 125
Diffussion Planner
Habitual Inforence (HI)
180
18
1.00
State
t- 150
- 175
- 200
t 225
250
t 275
Figure 9. Action distribution visualization Kitchen, plotted the same way Figure 5. The actions dimension-redueed by PCA.
17

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Dillusion Planner
140
140
335
175
805
0.7
939
359
039
990
329
500
sco
040
0.00
-33
-313
-353
-..0
Habitual Inference (III)
k25
815
220
520
1:0
0.0
-325
1335
-325
-025
-0.5
-339
-353
-359
-159
-030
-000
-A1
-375
-015
-33
-07
-075
-55
83
State
50
1=
100
Diffusion Planner
1.5
6.00
6.0
8.6
6.50
Habitual Inference (HI)
1.5
1.0
1.00
6.5
650
6.00
6.0
630
(20
022
625
125
6.0
130
-0.0
-.00
State
t= 150
== 175
t= 200
t= 225
t= 250
t= 275
Figure 10. Another example of action distribution visualization on Kitchen, plotted the same way as Figure 5, The actions are dimension- reduced by PCA.
18

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Dillusion Planner
Habitull Inference (III)
C:3
+:3
+:3
+:3
State
1=0
1=
1=
1=
75
100
1= 125
Diffusion Planner
Habitual Inference (HI)
+:3
+
+3
+
+:3
State
150
200
250
275
Figure Action distribution visualization AntMaze, plotted same way as Figure The actions are dimension-reduceed by PCA.
19

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Dillusion Planner
-44
-00
Habitual Inferencee (III)
-20
State
L=0
1=5
10
1= 15
20
1= 25
Diffusion Planner
-02
-0e
-35
Habitual Inference (HI)
-24
-04
State
30
t
t=
Figure Action distribution visualization MuloCo, plotted same way as Figure The actions are dimension-reduees by PCA.
20

---

Habitizing Diffusion Planning for Efficient and Effective Decision Making
Dillusion Planner
Habitual Inference (III)
333333
State
1=0
1 25
1= 50
1= 75
100
1= 125
Diffusion Planner
Habitual Inference (HI)
33333333
State
250 Figure 13. Another example of action distribution visualization Maze2D, plotted same way Figure 5,
150
200
21


Reviews:
[{"review_id": "0gOMJMytWh", "reviewer": "Reviewer_qHBM", "summary": "This paper proposes the Habi algorithm (a Diffusion Planner), which combines excellent performance with high-frequency inference speed. It utilizes a VAE-like inference framework to distill information from the diffusion planning process. Extensive experiments were conducted across various environments, achieving consistent improvements.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to your thoughtful comments and acknowledging our work \"***written very clearly / research question is very important / method makes sense / supported by extensive experimental validation / simple and elegant***\" as well as bringing the questions. We address your questions as follows.\n\n> Q1: GPU for decision-making speed tests.\n\nThanks for the suggestion. In fact, we already provided the results on both CPU and GPU in Appendix Table 4. We reported the results on CPU as CPUs are more portable (as descent CPUs can be put into mobile phones), thus be more practical when considering local deployment of decision models on individual robots and edge devices.\n\n> Q2: Decision frequency of DiffuserLite\n\nWe also noticed this. DiffuserLite serves as a key baseline for decision-making acceleration. We are unsure if absolute value differences stem from hardware variations, as we follow its [official code](https://github.com/diffuserlite/diffuserlite.github.io) for testing. In our experiments, we used consistent hardware (Apple M2 Max, Nvidia A100, AMD EPYC 7V13 64-Core) and software (PyTorch=2.2.2, numpy=1.22.4) to ensure fair and consistent speed test results. Therefore, the relative decision frequency presented in our paper could fairly reflects the differences in decision speed among various generative decision-making methods.\n\n> Q3: Relationship with world model and MBRL\n\nThank you for the suggestion. We treat Habi purely as an acceleration framework for generative decision-making. Its training process does not involve Q-Learning, nor does it interact with virtual reward dynamics. Habi's goal is to maintain the performance of the original generative model while accelerating decision-making within 1-2 hours of Habitual Training. Indeed, Habi is not limited to accelerating model-based planners (e.g., Diffuser, DV) but can also accelerate model-free planners (e.g., IDQL, DQL). We appreciate your suggestion and will discuss Habi's connection to offline MBRL methods (e.g., MOREC) in the related works section.\n\n> Q4: Can the ideas of habi be applied to more realistic robot control or robot control environments?\n\nYes, and thanks for suggesting extensive benchmarks. Beyond the Franka robotic arm manipulation tasks, we  evaluate Habi on the Adroit environment (including opening the door, driving the nail, repositioning the pen orientation, and relocating the ball) for dexterous manipulation assessment.\n\n| Robotics Environments | Diffusion Planner [1] | Habi |\n|--------------|------------------------|------|\n| Door | 104.2 ± 0.7 | 105.6 ± 0.3 |\n| Hammer | 124.4 ± 1.8 | 129.4 ± 0.4 |\n| Pen | 122.2 ± 1.8 | 121.0 ± 2.4 |\n| Relocate | 109.3 ± 0.5 | 108.8 ± 0.5 |\n| Franka Kitchen-M | 73.6 ± 0.1 | 69.8 ± 0.4 |\n| Franka Kitchen-P | 94.0 ± 0.3 | 94.8 ± 0.6 |\n\nThe results demonstrate that Habi maintains comparable performance to diffusion planners across various robot control tasks. In some environments like Door, Hammer, and Kitchen-P, Habi even shows slight improvements. We believe this highlights the robustness of our simple-yet-effective method and its potential for real-world robotic applications.\n\n> Q5: Why is the diffusion planner algorithm rarely applied in real robotic control scenarios? As far as I know, most mainstream algorithms are based on diffusion policy.\n\nWe first would like to make a disambiguity statement about the \"diffusion policy\" in your question. \n\n\n***--If you meant the paper Diffusion policy: Visuomotor policy learning via action diffusion from Chi et al.***\n\n\nThe premise of this question contains a misconception. Diffusion Policy is actually built upon Diffuser (a classic diffusion planner), with slight modifications to action modeling and visual inputs. Consequently, ***diffusion planner algorithms have indeed been deployed in robotics***, with DP being among the first to bridge the ML-robotics gap and demonstrate results on physical robots.\n\nFurthermore, the subsequent development directions of these two approaches differ: the robotics community typically emphasizes ***Imitation Learning*** with ***expert demonstrations***, while the ML community focuses on ***Offline RL*** that can derive optimal policies from ***varied-quality data***. The core algorithms, however, share the same diffusion-based foundation.\n\n\n***--If you meant the model-free diffusion policies such as Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning from Wang et al.***\n\nWe believe one of the most crucial problems of diffusion planners are its heavy computation cost even on a descent GPU: As we have shown in Appendix Table 4, Diffuser take 0.3~0.6 second for one decsion, which is not acceptable to real-world robot. However, research shows diffusion planners can outperform diffusion policies on many tasks [1]. This is why our work focused on accelarating diffusion planning while maintaining its effectiveness.\n\n[1] What Makes a Good Diffusion Planner for Decision Making? Lu et al. ICLR 2025", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "I think some mbrl work should be discussed, see weakness.", "other_strengths_and_weaknesses": "pro:\n- The paper is written very clearly, and the research question is very important. I quite agree that decision speed is a critical issue in the application of diffusion models for decision making.\n- The method makes sense and is supported by extensive experimental validation. It compares a series of strong baselines, such as diffusion planners and diffusion policies.\n- The method is simple and elegant, and it is expected to achieve both efficiency and high performance.\n\ncons:\n- Why not use a GPU for decision-making speed tests? Why does the decision frequency of DiffuserLite seem to be significantly lower than the paper version?  \n- Although the article describes the core idea using habitual decision-making, is the overall concept closer to some world model and representation-related works, such as TDMPC, TDMPC2, and MRQ (Towards General-Purpose Model-Free Reinforcement Learning)? By achieving better representations, it enables the distillation of the diffusion planner, thereby achieving better performance. From this perspective, Habi is similar to a more advanced offline MBRL method. Could it be discussed in relation to advanced offline MBRL methods? For example, MOREC (Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning).\"  \n- Can the ideas of habi be applied to more realistic robot control diffusion policy or robot control environments (such as libero, rlbench, etc.)?\n- Why is the diffusion planner algorithm rarely applied in real robotic control scenarios? As far as I know, most mainstream algorithms are based on diffusion policy.", "other_comments_or_suggestions": "see above", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ys86hpoXen", "reviewer": "Reviewer_JDXx", "summary": "This paper has introduced a simple yet effective framework to speed up Diffusion-based planners (Habi). During training Habi learns:\n- A prior encoder for context (state)\n- A Posterior encoder and decoder for distilling learned planning in diffusion.\n- A critic for evaluating actions. \nI like the elegant idea and strong performance, while there are a few questions/claims that I hope to discuss during the rebuttal period.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful comments and acknowledging \"***Simple yet effective approach / Well-motivated idea / Well written paper / elegant idea and strong performance***\". We will address all of your concerns as follows.\n\n> Q1: Does the expert/teacher planner have to be a diffusion-based model? What's the relationship between the proposed framework and diffusion-planners?\n\nGood point! Theoretically, Habi also supports acceleration for other ***probabilistic generative*** decision-making models. However, Habi is especially well-suited for diffusion planners for several important reasons:\n\n1. Many other generative models (like VAEs or flow-based models) are already computationally efficient, but often face performance limitations rather than speed constraints [1, 2]. Their bottleneck, however, is typically quality of decisions, not decision frequency.\n2. Diffusion models have emerged as the most powerful and well-performing probabilistic generative models in decision-making, as evidenced by the impact of Diffuser and related work. However, diffusion planners are inherently slow during inference due to their multi-step denoising process, which limits their practical application in real-world scenarios.\n\nHabi addresses this crucial gap between 1) and 2) by accelerating diffusion-based decision-making models while maintaining nearly lossless performance. To our best knowledge, Habi is the first framework to successfully preserve diffusion models' superior performance while achieving decision speeds comparable to traditional generative models.\n\nWe appreciate your suggestion and will clarify this important distinction in the revised paper.\n\n> Q2: The claim in L107-108 is a bit too strong:\n\"Habi can be used straightforwardly for any diffusion planning and diffusion policy models.\" While it seems that in the experiments the authors have only implemented HI for one base planner (correct me if I misunderstood).\n\nActually, we used Habi on two base diffusion decision making models: [1] and [3] (line 738, Appendix Table 3). We selected these two models since they are best performing on corresponding benchmarks. \n\nHowever, we greatly appreciate this suggestion since we did not realize this clarity issue. We will make more clear statements in the main texts of the revised paper.\n\n> Q3: Why is the framework using ztq instead of ztp for training the Critic?\n\nThe Prior distribution and Posterior distribution are constrained through KL divergence. Due to the nature of the KL constraint KL(q||p):\n\n$$KL(q(z) \\parallel p(z)) = \\int q(z) \\log \\frac{q(z)}{p(z)} dz$$\n\nThe prior $p(z)$ is required to cover the posterior distribution $q(z)$, but points with low probability in the posterior may still be sampled by the prior. Using these low-probability samples to train the critic would introduce noise and potentially mislead the training process.\n\nTherefore, using latents generated from the posterior can reduce the impact of these misleading samples on critic training, ensuring more reliable and accurate critic learning. Intuitively, this shares a roughly similar idea with *Teacher Forcing* in autoregressive models: During training, instead of feeding the model's own predicted output (akin to prior z in Habi), teacher forcing uses the ground-truth (akin to posterior z in Habit) to prevents compounding errors. \n\nWe also make an additional experiment on maze2d to empirically validate this design choice:\n\n|Environment| Posterior Critic | Prior Critic |\n|----|----|----|\n|Maze2D-Large| 199.2 ± 2.0 | 195.3 ± 2.2 |\n|Maze2D-Medium| 150.1 ± 1.5 | 150.6 ± 1.5 |\n|Maze2D-Umaze| 144.3 ± 1.7 | 143.0 ± 1.7 |\n|**Average**| **164.5** | 163.0 |\n\nIt can be seen that using posterior z for critic training works slightly better.\n\nThank you again for your acknowledgement of Habi and your contribution to our paper. Hope that we have address your concerns.\n\n**Reference**\n\n[1] What Makes a Good Diffusion Planner for Decision Making? Lu et al. ICLR 2025\n\n[2] CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Making, Dong et al. NeurIPS2024\n\n[3] Diffusion policies as an expressive policy class for offline reinforcement learning, Wang et al. ICLR 2023\n\nThank you again for your constructive feedback! We hope the responses above have addressed all your comments. Please kindly let us know if you have additional suggestions, and we would be more than happy to discuss them.", "claims_and_evidence": "- Does the expert/teacher planner have to be a diffusion-based model?\n\nFrom the framework, I didn't see any special assumptions on the pretrained planner, and it seems to me this should work for any learned model-based RL algorithm. What's the relationship between the proposed framework and diffusion-planners?\n\n- The claim in L107-108 is a bit too strong:\n\n\"Habi can be used straightforwardly for any diffusion planning and diffusion policy models.\" While it seems that in the experiments the authors have only implemented HI for one base planner (correct me if I misunderstood).", "methods_and_evaluation_criteria": "The method and evaluation make sense for the problem and application.", "theoretical_claims": "N/A\n\nThere is only a (widely known) ELBO theoretical proof for VAE shown in the supplementary.", "experimental_designs_or_analyses": "This is one design choice that I hope to see some analyses:\n- Why is the framework using $z_t^q$ instead of $z_t^p$ for training the Critic?\n\nIt seems to me that during inference, $z_t^p$ is the one used for Critic evaluation, so using $z_t^p$ during training seems to be a more intuitive design, is there any theoratical/empirical support for such design?", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- Simple yet effective approach\n- Well-motivated idea\n- Well written paper\n\nWeaknesses:\n- Some claims are too strong to be supported in the experiments.\n- It is unclear to me about the relationship between this approach and diffusion.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B11EhGdfe7", "reviewer": "Reviewer_ybn9", "summary": "This paper presents Habi, a framework that habitizes diffusion planning into faster decision-making models by using a VAE-based approach inspired by biological habit formation. While the method demonstrates impressive speedups and maintains comparable performance to diffusion planners, the technical innovation is limited as it primarily applies standard VAE techniques to policy distillation without substantial modifications. Additionally, the claimed advantages over direct distillation are marginal in several tasks and lack thorough component-wise analysis. I have some concerns about the experimental setup and fairness of comparisons, as Habi fundamentally depends on pre-trained diffusion planners and should be more accurately positioned as an \"acceleration technique\" rather than a standalone decision-making algorithm.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your detailed and comprehensive comments, the following responses are to address your concerns.\n\n>Q1: Position Habi as an \"acceleration technique\" rather than a standalone algorithm.\n\nThank you for the clarification. Actually, we indeed position Habi as a general acceleration framework for diffusion planners in this paper. Our goal was creating an elegant framework that significantly improves decision frequency while maintaining effectiveness on SOTA diffusion planners through brief habitual training (1-2 hours).\n\nOur focus is on decision-frequency, with performance comparisons to demonstrate Habi's effectiveness.\n\n>Q2: Comparisons with established distill methods (e.g., soft targets)\n\nDiffusion models' probability distributions aren't directly accessible, making traditional distillation methods using cross-entropy loss (e.g., soft targets) inapplicable. This motivated Habi's development.\n\nWe included comprehensive comparisons with two categories of acceleration approaches: numerical acceleration methods (DiffuserLite) and distillation-like frameworks (DTQL). The Direct Distill baseline uses the same architecture as Habi but performs imitation learning directly on state-action pairs.\n\n>Q3: Performance improvements seem marginal for simple tasks (Maze2D, MuJoCo)\n\nThere is probably a misunderstanding. Our rigorous evaluation methodology (5 training seeds × 500 evaluation seeds) demonstrates these improvements are **_statistically significant_** and reproducible, which is supported by:\n\n$$\\mu_{avg} = \\frac{1}{G} \\sum_{i=1}^G \\mu_i$$\n\n$$stderr_{avg} =\\frac{\\sqrt{\\frac{1}{G}\\sum_{i=1}^G(\\mu_i^2 + N_{seed} (stderr_i)^2) - \\mu_{total}^2}}{\\sqrt{G*N{seed}}},$$\nwhere $G$ is the task number, $N_{seed}$ is the number of seed.\n\n**Performance Comparison**:\n\n|Task|Method|HI(Ours)|HI w/o Critic|Direct Distill|\n|--|--|--|--|--|\n|MuJoCO|Absolute|89.24±0.29|87.46±0.32|84.98 ± 0.26\n|-|Δ vs Ours|-|-1.78 (p=1.88e-5)|-4.26 (p<1e-10)|\n|Maze2D|Absolute|164.53±1.07|161.33±1.11|159.87±1.10\n|-|Δ vs Ours|-|-3.20 (p=0.019)|-4.66 (p=0.001)|\n\nMoreover, the improvements becomes more pronounced in ***complex environments***: in Kitchen and Antmaze domains, Habi exhibits ***19.1–29.4%*** stronger performance.\n\n>Q4: Necessity of Critic.\n\nThank you for your question. The critic is essential. Without it, performance in complex environments like Antmaze suffers an 18.5% degradation. Tested with over 500 seeds, the critic consistently provides performance improvements. Meanwhile, the critic is a shallow MLP that doesn't add significant computational burden.\n\n> Q5: Questions on network structures, diffusion planners, and latent dimensionality.\n\nThank you for your suggestion. For high decision speed, MLPs are already sufficiently simple. DV and DQL adequately represent the two types of diffusion planning. Regarding latent dimensionality, our experiments show:\n\n|Env|dim(z)=64|128|256|512|\n|--|--|--|--|--|\n|Antmaze-L-D|62.2±2.1|60.7±2.0|65.2±2.0|70.6±2.0|\n|Antmaze-L-P|79.3±1.8|80.3±1.7|81.7±1.7|85.9±1.5|\n|Antmaze-M-D| 88.5±1.4|89.6±1.3|88.8±1.4|78.6±1.8|\n|Antmaze-M-P| 78.0±1.8|82.3±1.7|85.3±1.5|86.7±1.5|\n|**Avg**|77.0|78.2|80.3|80.5|\n|Maze2D-L|204.0±1.9|204.3±1.9|199.2±2.0|202.4±1.9|\n|Maze2D-M|151.1±1.5|151.8±1.4|150.1±1.5|149.2±1.6|\n|Maze2D-U|143.6±1.7|144.5±1.7|144.3±1.7|144.1±1.7|\n|**Avg**|166.2|166.9|164.5|165.2|\n\nLatent dimension has **_modest impact_** on performance. For Antmaze, there's slight improvement as dimension increases. For Maze2D, performance remains stable. As our method does not involve an information bottleneck, we selected dim(z)=256 as a balanced choice providing good performance without unnecessary computational overhead.\n\n>W1: Innovation & Q5: Difference between Habi and VAE\n\nWe understand your concerns. As acknowledged by other reviewers (***qHBM***, ***JDXx***), Habi maintains simple and elegance for easy using, while Habi and VAE are different fundamentally:\n\n**Function**: VAE's latent dimension is smaller than input dimension, forming an information bottleneck; Habi uses a large latent dimension (256) as it doesn't target compact representation.\n\n**Latent Bottleneck**: VAE's latent dimension is smaller than input dimension, forming an information bottleneck; Habi uses a large latent dimension (256) as it doesn't target compact representation.\n\n**Learnable Prior**: VAE employs a fixed unit-Gaussian prior; Habi's prior distribution conditions on the current state and is learned.\n\n>W2. biologically-inspired rather than technical innovation\n\nThanks for the suggestion. While the high-level idea of our paper is brain-inspired, we agree that we should more clearly highlight the technical innovations by revising the paper.\n\n>W3. Lack of real-world, vision-based deployment.\n\nThank you for the suggestion. We acknowledge it in Section 6 of our paper. This work focuses on algorithmic contributions using standard offline RL for robust benchmarking in the simulation, which is common practice in the field.", "claims_and_evidence": "1. I have some concerns about the experimental setup and fairness of comparisons. The paper frames Habi as a complete decision-making method, but its fundamental nature is a two-stage process that depends on pre-trained diffusion planners. This dependency is not adequately acknowledged, creating a misleading comparison. Habi should be more accurately positioned as an \"acceleration technique\" rather than a standalone decision-making algorithm. This would better reflect its true nature and enable more appropriate comparisons with other acceleration approaches.\n2. For a truly fair evaluation, the paper should provide comprehensive cost analyses that include both the initial planner training and subsequent habitization process, giving readers a complete understanding of the total computational investment required. Moreover, the authors claim superiority over standard distillation approaches, but evidence is unconvincing. In several tasks (MuJoCo and Maze2D), performance improvements are marginal (only 2-5% better). Without comprehensive component-wise analysis, it's unclear what drives these improvements or whether they are statistically significant.", "methods_and_evaluation_criteria": "1. The comparision and the experiment setup is not convicing. The comparison between Habi (which utilizes pre-trained planners) and methods that learn from scratch (like AdaptDiffuser) is fundamentally imbalanced.(See Claims Q1 and Q2)\n2. The \"Direct Distill\" baseline lacks detailed implementation specifications, making it impossible to verify whether it represents state-of-the-art distillation techniques. Comparisons with established knowledge distillation methods (e.g., Hinton's approach with soft targets) are conspicuously absent.\n3. The paper introduces a Critic component for action selection, but Table 5 shows that in many environments, performance without the Critic (N=1 case) is already strong. This raises questions about the necessity of this additional component and complicates the architecture without clearly justified benefits.", "theoretical_claims": "1. While the habitization process draws inspiration from cognitive science, the theoretical mapping between brain processes and the VAE framework is superficial. The paper does not sufficiently establish why ELBO optimization should effectively model the transition from goal-directed to habitual behavior.\n2. The core technical contribution is essentially applying a standard VAE for policy distillation. The ELBO loss (L = L_recon + β_KL·L_KL) comes directly from standard VAE theory with minimal modification. The paper reinterprets this as \"habitization\" without substantial theoretical innovation.", "experimental_designs_or_analyses": "1. The comparision and the experiment setup is not convicing.(See Claims Q1 and Q2)\n2. While the paper includes analysis on the number of candidate samples (N), it lacks comprehensive ablations for other critical components such as network architecture choices, latent space dimensionality, and the impact of different diffusion planners as teachers.", "relation_to_broader_scientific_literature": "See above.", "essential_references_not_discussed": "See above.", "other_strengths_and_weaknesses": "1. Innovation largely limited to applying existing techniques (VAE, critic-based selection) in a new context\n2. Positioning as a biologically-inspired method appears to be primarily a narrative framing rather than a substantive technical innovation\n3. The lack of real-world deployment testing or vision-based tasks limits confidence in practical applicability", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4AYBcbp9NU", "reviewer": "Reviewer_6Hyb", "summary": "This paper introduces a novel and general framework that can accelerate the existing diffusion-based planning models. The motivation is that, most of the diffusion-based planning methods are very slow due to the iterative denoising steps during deployment. In this work, a VAE-like learning framework is proposed to learn the distilled policy from the pre-trained diffusion models. The framework contains a prior encoder (state as input), a posterior encoder (state, action as input) and a latent decoder based on the posterior latent. The typical MSE reconstruction loss is used to distill the diffusion policy to the VAE-policy, and an extra KL-divergence loss is devised to align the decision space between prior and posterior encoders. A critic is trained to evaluate the quality of the samples. And during inference, out of the generated samples, the one with the highest critic score will be selected for deployment. The authors conduct extensive experiments under D4RL benchmark and compare with deterministic policies, diffusion policies/planners and accelerated diffusion-based methods. The proposed method demonstrates a much better computation efficiency with similar solution quality as the strongest baselines.\n\n## update after rebuttal\nI have read the author's response as well as other reviewers' comments. Overall, I think the work is novel and is okay to be accepted by ICML, though a few flaws exist. I updated my final ratings as \"weak accept.\"", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank for reviewing our paper and acknowledging \"***idea is very straightforward and novel / easy to read / extensive experiments / reasonable ablation studies.***\" We believe the following response can address your concerns.\n\n\n> Q1: Comparison with other generative decision-making baselines\n\nThank you for your suggestions. Since references 2 and 4 involve visuomotor policies and lack the same benchmarks, we compared our method with:\n- Flow to Better: Offline preference-based reinforcement learning via preferred trajectory generation\n- Boosting continuous control with consistency policy\n\nThe policy of FTB is also deterministic, which is the same as BC/SRPO.\n\n| Environment | HI (Ours) | FTB | CPIQL | CPQL |\n|-------------|-----------|-----|-------|------|\n| HalfCheetah-ME | **98.0 ± 0.0** | 85.2 ± 0.7 | 81.0 ± 1.7 | 97.8 ± 0.5 |\n| HalfCheetah-MR | **48.5 ± 0.0** | 38.4 ± 1.3 | 48.0 ± 1.4 | 46.6 ± 0.8 |\n| HalfCheetah-M | 53.5 ± 0.0 | - | 54.6 ± 1.0 | **56.9 ± 0.9** |\n| Hopper-ME | 92.4 ± 2.0 | **111.1 ± 2.0** | 110.6 ± 1.4 | 110.4 ± 3.2 |\n| Hopper-MR | **102.0 ± 0.0** | 89.6 ± 4.9 | 100.6 ± 1.5 | 97.7 ± 4.6 |\n| Hopper-M | **102.5 ± 0.1** | - | 99.7 ± 2.0 | 99.9 ± 4.5 |\n| Walker-ME | **113.0 ± 0.0** | 109.3 ± 0.3 | 110.9 ± 0.2 | 110.9 ± 0.1 |\n| Walker-MR | **102.0 ± 0.0** | 79.1 ± 1.4 | 91.8 ± 2.8 | 93.6 ± 5.6 |\n| Walker-M | **91.3 ± 0.1** | - | 86.2 ± 0.6 | 82.1 ± 2.4 |\n| **Tasks Average** | **89.2** | - | 87.0 | 88.4 |\n| **Shared Tasks (w/o M dataset) Average** | **_92.7_** | 85.5 | 90.5 | **92.8** |\n| **Frequency** | 1329.7 | **11892.6** |  578.2 | 570.1 |\n\nWe conducted comparisons on common MuJoCo locomotion tasks (since these work did not present the results on the other benchmarks in our paper, and FTB lacks the results on Medium dataset). Habi achieves the best performance in most tasks and demonstrates a higher overall score across environments.\n\nImportantly, we do not want to convey the message that Habi (variational Bayesian method) is better than flow-based method. Indeed, they are **orthogonal** and can be combined (e.g., Kingma et al., 2016. \"Improved Variational Inference with Inverse Autoregressive Flow\"). We appreciate your question and will have a deeper dive into potential collaboration between flow-based method in our future work.\n\n\n> Q2: The proposed method requires a pre-trained diffusion model and also needs to train a critic - which requires more time in training than other \"train-from-scratch\" methods.\n\nThank you for your question. This work positions Habi as a general acceleration framework designed to speed up given diffusion planners. Our starting point is to maintain diffusion planning performance while significantly improving decision frequency. \n\nIndeed, Habi's training process is light-weighted,e.g., Habi only takes around 2 hours for Habitual Training using one A100 GPU. \n\n> Q3: Though the critic can improve per-step quality, the overall episode return does not seem to be improved much and even degrades later.\n\nThere is probably a misunderstanding. We assume you are talking about Figure 7 (please let us know if not the case). Figure 7 illustrates the impact of increasing candidates on performance given the same critic. When there are too many candidates, performance may decline. Therefore, around 5 candidates is recommended. This phenomenon occurs because too many candidates amplify the critic's judgment errors for individual actions.\n\n\n> Q4: Why not report the computation time for BC and SRPO baselines?\n\nWe acknowledge that BC and SRPO are super fast, as they used deterministic policy function with single-pass inference (i.e., an MLP). We did not report the computational time since the bottleneck of inference time no longer come from action computation, but rather from other parts such as `env.step(action)`. Thanks for the advice, and we will revise the paper to clarify this.\n\nHowever, probabilistic generative models like diffusion policies and planners typically achieve superior performance by modeling complex, multi-modal action distributions -- but at significant computational cost. Our work focuses specifically on accelerating probabilistic generative decision-making methods while maintaining their performance advantages over deterministic approaches. Habi addresses this trade-off by accelerating probabilistic generative models while preserving their stochastic nature and performance benefits.\n\n\nThank you again for your constructive feedback! We hope the responses above have addressed all your comments. Please kindly let us know if you have additional suggestions, and we would be more than happy to discuss them.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes (although there is no proof in this paper).", "experimental_designs_or_analyses": "Yes. I checked all the experiments.", "relation_to_broader_scientific_literature": "This paper proposed diffusion model framework can work at very high frequency, opening the door for fast online deployment for systems that require real-time performance.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Pros:\n* The idea is very straightforward and novel.\n* The paper is easy to read.\n* Extensive experiments have been conducted (including reasonable ablation studies).\n\nCons:\n* Missing vital baselines: should compare with flow-based methods [1, 2], vanilla VAE, and consistency models [3, 4].\n* The proposed method requires a pre-trained diffusion model and also needs to train a critic - which requires more time in training than other \"train-from-scratch\" methods.\n* Though the critic can improve per-step quality, the overall episode return does not seem to be improved much and even degrades later.\n\nReferences:\n[1] Zhang, Zhilong, et al. \"Flow to better: Offline preference-based reinforcement learning via preferred trajectory generation.\" The Twelfth International Conference on Learning Representations. 2023.\n[2] Zhang, Qinglun, et al. \"FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulation.\" arXiv preprint arXiv:2412.04987 (2024).\n[3] Chen, Yuhui, Haoran Li, and Dongbin Zhao. \"Boosting continuous control with consistency policy.\" arXiv preprint arXiv:2310.06343 (2023).\n[4] Prasad, Aaditya, et al. \"Consistency policy: Accelerated visuomotor policies via consistency distillation.\" arXiv preprint arXiv:2405.07503 (2024).", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "Why not report the computation time for BC and SRPO baselines?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a method for accelerating diffusion-based planners. The reviewers appreciated the importance of the problem, the simplicity of the method, and the strength of the results. They also found the paper well written and well motivated. The main concerns (missing baselines, clarifying the computational costs) were addressed during the rebuttal period, alongside a number of other concerns. After the rebuttal and a discussion among the reviewers, all four reviewers agreed that the paper should be accepted. I am therefore recommending that the paper be accepted.

My expectation is that authors will incorporate the discussion and new experiments from the rebuttal into the camera ready version of the paper. I'd also encourage them to clarify the details for the "Direct Distill" baseline, as suggested by Reviewer 6Hyb.
Decision: Accept (poster)