Model-based Diffusion for Trajectory Optimization
Authors: Chaoyi Pan
Zeji Yi
Guanya Shi
Guannan Qu
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion
Trajectory Optimization
Motion Planning
Robotics
Sampling-based Control
Primary area: diffusion_based_models
TL;DR: Model-Based Diffusion (MBD) solves trajectory optimization by using model information for score computation.

Abstract: Recent advances in diffusion models have demonstrated their strong capabilities in generating high-fidelity samples from complex distributions through an iterative refinement process. Despite the empirical success of diffusion models in motion planning and control, the model-free nature of these methods does not leverage readily available model information and limits their generalization to new scenarios beyond the training data (e.g., new robots with different dynamics). In this work, we introduce Model-Based Diffusion (MBD), an optimization approach using the diffusion process to solve trajectory optimization (TO) problems without data. The key idea is to explicitly compute the score function by leveraging the model information in TO problems, which is why we refer to our approach as model-based diffusion. Moreover, although MBD does not require external data, it can be naturally integrated with data of diverse qualities to steer the diffusion process. We also reveal that MBD has interesting connections to sampling-based optimization. Empirical evaluations show that MBD outperforms state-of-the-art reinforcement learning and sampling-based TO methods in challenging contact-rich tasks. Additionally, MBD’s ability to integrate with data enhances its versatility and practical applicability, even with imperfect and infeasible data (e.g., partial-state demonstrations for high-dimensional humanoids), beyond the scope of standard diffusion models. Videos and codes are available in the supplementary materials.

Model-Based Diffusion for Trajectory Optimization
Chaoyi Pan', Zeji Yi', Guanya Shi', Guannan Qu
Carnegie Mellon University
{chaoyip, zejiy, saaayyyyggngaasss.....aaaaaaaaaa guanyas, Gandrew. cmu cdu.edu
Abstract
Recent advances in diffusion models have demonstrated their strong capabilities generating high-fidelity samples from complex distributions through an iterative refinement process. Despite the empirical success of diffusion models motion planning and control, the model-free nature of these methods does not leverage readily available model information and limits their generalization to new scenarios beyond the training data (e-g.. new robots with different dynamics). In this work, we introduce Model-Based Diffusion (MBD), an optimization approach using the diffusion process to solve trajectory optimization (TO) problems without data. The key idea is to explicitly compute the score function by leveraging the model information in TO problems, which why we refer our approach model- based diffusion. Moreover, although MBD does not require external data, it can be naturally integrated with data of diverse qualities to steer the diffusion process. We also reveal that MBD has interesting connections to sampling- based optimization. Empirical evaluations show that MBD outperforms state-of-the of-thhe-rt reinforcement learning and sampling-based TO methods in challenging contact-rich tasks. Additionally, MBD's s ability to integrate with data enhances its versatility and practical applicability, even with imperfect and infeasible data (e.g., partial-state state demonstrations for high- bigh--imensional humanoids), beyond the scope standard diffusion models. Videos and codes; https: httpp:///ii :/lpsa///ccar lab. github. 1o Lo/mbd/
-based
Introduction
Trajectory optimization (TO) aims to optimize the state and control sequence to min- imize a cost function while subject specified dynamics and constraints. Given non- linear, non-smooth dynam- ics and non-convex objec- tives and constraints, tradi- tional optimization methods like gradient-based methods and interior point methods are less effective in solving TO problems. In response, diffu- sion models have emerged as a powerful tool for trajectory generation in complex dy- namical systems due to their expressiveness and scalabil- ity 12 54
Forward
yo
Vayi-n + vI aw
Ap
ydーD
MON1
MY
B--1
XT
Model information M.
Backward yo-1)
y + (1 agy polog prring
Model-based score approximation
Model-based diffission humanoid running tasks Figure 1: MBD refines the trajectory by leveraging the dynamics model directly without relying on demonstration data.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Although diffusion models excel when learning from sargesscale, bigg-dimensional. and high-quaie -ualitya demonstrations, their dependency on such data limits their practicality. For example, after training a manipulation task with a specific robotic arm, the model may struggle to generalize to new tasks with a different arm as the underlying dynamies change. This limitation arises from the model-free nature of existing diffusion-basel based methods, which do not leverage readily available model information to enhance adaptability. Moreover, existing diffusion-bant approaches often require high-quality (in terms of optimality and feasibility) demonstration data, which limits their applications in various scenarios with imperfect data, such as dynamically infeasible trajectories (e.g. generated by high-level planners using simplified models) and partial demonstrations (e.g., lower-body-onfe demonstrations for a high-ding digendimensional humanoid).
Fortunately, unlike diffusion model's applications in vision or language where data is from unknown distributions (e.g., internet-scale image data), in trajectory optimization, we often know the distri- bution of desired trajectories. which is described by the optimization objectives, constraints, and the underlying dynamics model, although such a distribution is intractable to directly sample from. Diffusion models offer tantalizing new perspective, by iteratively refining samples from isotropic Gaussians to meaningful desired distributions in manageable steps, rather than directly learning the complex desired distribution. Inspired by this, we propose Model-Based Diffusion (MBD) that utilizes model information to approximate the gradient of the log probability density function (a.k.a. score function) and uses it to iteratively refine sampled trajectories solve TO problems, depicted in Fig. []] This model-centric strategy allows for the generation of dynamically feasible trajectories in data-free manner, and gradually moves them towards more optimal solutions. Furthermore, by using demonstrations as observations of the target distribution, MBD) can be smoothly combined with data of different qualities to steer the diffusion process and enhance its effectiveness. Particularly, we merge the demonstration data into the sampling process by evaluating their likelihoods with the model and use them to improve the estimation of the score function. Our contributions are threefold:
We introduce the Model-Based Diffusion (MBD) framework for trajectory optimization, utilizing the dynamics model to estimate the score function. This enables an effective trajectory planner given non-smooth dynamics and non-convex objectives, such as contact-rich manipulation tasks or high-dimensional humanoids.
Our analysis and empirical evaluations demonstrate that MBD matches, and often exceeds, the performance of existing reinforcement learning and sampling-based TO methods. In particular, MBD outperforms PPO by 59% in various tasks within tens of seconds of diffusing.
We demonstrate MBD's s flexibility in utilizing diverse imperfect data to steer the diffusion process and further enhance the performance. Specifically, the resulting whole-body humanoid trajectory from MBD is more natural by utilizing the swwer-odd--tttt-onn MBD can effectively address long-horizon sparse-reward Umaze navigation tasks by leveraging infeasible demonstrations generated by an RRT planner with simplified dynamics.
human motion data. Similarly,
2 Related Work
Diffusion Models.
Diffusion models have been widely adopted generative models for high- dimensional data, such as image [51, audio [33, and text [8] through iterative refinement pro- cesses [50 28). The backward process can be viewed as gradient prediction [52] or score match- ing [53]. which learns the score function to move samples towards the data distribution. We deliver new methods perform the backward diffusion process using the available model information.
Sampling-based Optimization. Optimization involving black-box functions is widely applied across various fields, including hyperparameter tuning and experimental design [49 27). Evolutionary algorithms like CMA-ES are often used to tackle black-box optimization problems, dynamically modifying the covariance matrix to produce new samples (24]. Such problems can also be efficiently addressed within the Bayesian optimization framework [50 19,, which offers greater efficiency. Nonetheless, traditional BO algorithms are generally restricted to fow-dimensional problems.
Trajectory Optimization. Traditionally, trajectory optimization (TO) is solved using gradient- based optimization, which faces challenges such as non-convex problem structures, nonlinear or discontinuous dynamics, and high-dimensional state and control action spaces. As two equivalent formulations, direct methods (25] and shooting- based methods [29 commonly used to solve TO

---

problems, where gradient-based optimizers such as Augmented Lagrangian [2), Interior Point [36. and Sequential Quadratic Programming 48 are employed. To leverage the parallelism of modern hardware and improve global convergence properties, sampling-hases methods like Cross-Entropy
Motion Planning (CEM) 3 and Model Predictive Path Integral (MPPI) [58] 62 have been proposed to solve TO by sampling from target distributions. To solve stochastic optimal control problems, trajectory optimization has also been framed as an inference problem in a probabilistic graphical model, where system dynamics defines the graph structure [35 [3). This perspective extends methods such as iLQG by integrating approximate inference techniques to improve trajectory optimization (55). The connection between diffusion and optimal control has been explored in [10], which motivates us to use diffusion models as solvers for trajectory optimization.
Diffusion for Planning. Diffusion- planners have been used to perform human motion gener- ation 12 and multi-agent motion prediction (34). Diffusion models are capable of generating complete trajectories by folding both dynamics and optimization processes into a single framework, thus mitigating compounding errors and allowing flexible conditioning 33
40 5].
In addition, they
have been adeptly applied to policy generation, enhancing the capability to capture multimodal demonstration data in high-dimensional spaces for long-horizon tasks [46 15]. These works assume no access to the underlying dynamics, limiting the generalization to new environments. To enforce dynamics constraints, SafeDiffuser [60] integrates control barrier functions into the learned diffusion process, while Diffusion- CCCsP [61] composes the learned geometric and physical conditions to guarantee constraint compliance. Our approach uses diffusion models directly as solvers, rather than simply distilling solutions from demonstrations.
Langevin- Markov Chain Monte Carlo for Global Optimization. Gradient- based sampling algorithms have been widely used in global optimization, where the energy function J is optimized by sampling from the Boltzmann distribution oc exp(-x) [57] 43) By annealing the temperature zero, the sampling process converges to the global minimum of the energy function J [31 The convergence of Langevin- -aween MCMC methods has been well studied in both continuous and discretized settings T6 21), showing that the distribution will converge probability to the target distribution with certain decreasing schedule of the step size and temperature A. In practice, the most common Langevin- based MCMC methods are unadjusted Langevin Monte Carlo (ULMC) 17 and Underdamped Langevin Monte Carlo (UdLMC) [14], with convergence rates of O 2 log and
O(+) given strongly convex and smooth energy functions, respectively. Recently, Langevin-based MCMC methods have been integrated into diffusion processes to improve global convergence and sampling efficiency (30, 6). where the score function is estimated by Monte Carlo to accelerate and stabilize the diffusion process. Our work differs from these methods in that we aim to sample from the proh-prrobbilityy region of the target distribution without assuming access the gradient of the energy function, and without assuming the energy function is smooth or convex.
3 Problem Statement and Background
Notations: We use lower (upper) scripts to specify the time (diffusion) step, e-g., Tt, Me ye represent the state, control and state-control pair at time t, and (+) represents the diffusion state at step i.
This paper focuses on a class of trajectory optimization (TO) problems whose objective is to find the sequences (xx) and (u:) that minimize the cost function J(r1:; un:r) subject to the dynamics and constraints. The optimization problem
can be formulated as follows:
min J (TITT = BTT,MIT
Σ t=0
4((x, ut)
(la)
s.t. Fu = Fimi
(1b)
T++1 fi
Vt = 0, 0,..,..,TT T
(1c)
(e 42) s O, Vt= 0, 1..., T.
,T 1.3
(1d)
where 24 Th and Ug "R the state and control time t, ft R n Th R represents the dynamics, ge : IR the x I + R are the constraint functions and 4 R R + RR are the stage
'We assume deterministic dynamics for simplicity to sample the dynamically feasible trajectory. The extension to stochastic dynamics is straightforward.
3

---

costs. We use Y = [21:2; W1rr] to denote all decision variables. Traditionally, TO is solved using nonlinear programming. which faces challenges such as non-convex problem structures, nonlinear or discontinuous dynamics, and high-dimensional state and control action spaces. Recently, there has been a growing interest bypassing these challenges by directly generating samples from the optimal
trajectory distribution using diffusion models trained on optimal demonstration data [12 40 46 61].
To use diffusion for TO, ① is first transformed into a sampling problem. The target distribution Po ()(Y)) proportional dynamical feasibility PY(Y) I (xx (1-1)))1; optimal- ity ps oc exp >) and the constraints (Y) oc I (Z u)) s 0), i.e.,
Po( pc(P)p)((pp(())
(2)
Obtaining the solution Y from the TO problem in Eq. is equivalent to sampling from Eq. given low temperature À O. In fact, in Appendix A.2 prove that the distribution of J(Y) with Y (.) converges in probability to the optimal value _ as x - O, under mild assumptions. However, it is generally difficult to directly sample from the high-dimensional and sparse target distribution (-). To address this issue, the diffusion process iteratively refines the samples following backward process, which reverses predefined forward process as shown in Fig. I The forward process corrupts the original distribution Po(-) to an isotropic Gaussian PN (-) by incrementally
adding small noise to and scaling it down by va, to maintain an invariant noise covariance (see Fig. b) for an example). Mathematically, this means we iteratively obtain Piji-1 P (-)((-)) ~ ((aay (1-as) I). Because the noise at each time step is independent, the conditional distribution of y (0 (t-1) also leads that of y ( (0))
y(i) ~ p:(-) with
(0)((11(0))~ N(V (1-a.)1),
â II ap.
(3)
The backward process P 11-1 ( is the reverse of the forward process P.. Pa (i-1)
which removes the noise from the corrupted distribution px()) to obtain the target distribution po)). The target distribution po (-) in the diffusion process is given by:
Pi-1 Pi-1 (y(=-1)) = ( )r (Y(())) ay
(4)
Po(r()))= (N(N))))
| Pz- (Y N
Standard diffusion models 133 40 61. which refer to Model-Free Diffusson (MFD), solve the backward process by learning score function merely from data. In contrast, we propose leveraging the dynamics model to estimate the score improve the generalizability of the model and allow natural integration with diverse quality data.
Model-Based Diffusion
In this section, we formally introduce our MBD algorithm that leverages model information to perform backward process. To streamline the discussion, in Section 4.1 we first present MBD with Monte Carlo score ascent to solve simplified and generic unconstrained optimization problems. In Section 4.2 extend MBD to the constrained optimization setting to solve the TO problem given complex dynamics and constraints. Lastly, Section 4.3 we augment the MBD algorithm with demonstrations to improve sample quality and steer the diffusion process.

---

4.1
Model-based Diffusion as Multi-stage Optimization
(a). Objective Function JO).
(b) Forward Density
(c) Backward Process
0.00
1o0ooo000 w
0.000
6.23
Objective
0.00
Density
0.015
F20
06
0.010
11.00
Reevero SDE
0000
Ascend
11.00
50 100
Difiusion Step
Figure 2: Reverse SDE vs. Monte Carlo score ascent (MCSA) on a synthetic highly non-convex ob- jective function. (a) Synthesized objective function with multiple local minima.
(b) The intermediate
stage density pt).). where peaked po(-) )) is iteratively corrupted to a Gaussian px (.) (c) Reverse SDE
vs. MCSA: Background colors represent the density of p.(.) at different stages. MCSA converges faster due to larger step size and lower sampling noise while still capturing the multimodality.
We consider the reverse process for an unconstrained optimization problem miny J(Y), where the target distribution is pa ())) oc exp (- (- ). our MBD framework, "model" implies that we can evaluate (() for arbitrary (()), enabling us to compute the target distribution up to the normalizing constant.
MBD uses Monte Carlo score ascent instead of the commonly adopted reverse SDE approach in MFD. Specifically, when denoising from log De ()) and then scales the sample by the factor ي as defined in the forward process:
1, MBD performs one step of gradient ascent on
y(-11) = 1 -((() (1-ā,) (ygp; Va,
(6)
Critically, with the model-informed po (Y("),, we can estimate the score function Vyco log (V(()) by connecting Pa (Y()) to (o))( via Bayes' rule:
Vyco (x(() Vycal f (H (()) (o()) dr I VycoPoO )ou (x) ()
(Y y(O) (P(()) dY ()O( ((0)) d
(7a)
y()-aar(())
1-41 Pio (o0))
(7b)
(V Y )ro (V()) dY(())
((i
Vor I y (Tp y(u)) dy(")
(7c)
1 1-a (Y(()) y (p(())) dy
Between Eq. 7a and Eq.
7b
we use the forward Gaussian density in Eq. 3): (Y (()))
(y() -y)-vary (a) (y (arr(
exp(-
). Its log-likelihood gradient is Vy (()) y()))
1-d; I-aa (Y(1) Va,Y ( ) (Y y(!)). Given (() as the integration variable in Eq. 7ch
2
Pao(Y (Y y()) is evaluated as a function of (0) parameterized by (()) Based on that, we define the function Φi (Y)))) as:
1 (r y(t) (y())
y(t)
y(i) I --1) (8)
(x((())) ()(()(() y)()) exp(
/a,
Va,
N
2
1-0:
04
a
5

---

This finding enables the Monte-Catee estimation for computing the score function. We collect a batch of samples from 6:(+) which we denote as yr(1) and approximate the score as:
y()) Vā. ( (e (V(0))0 dy
Vy logpi(V()))
(9a)
l- 1- ( (Y dY
y())
Von
Eyoryo V po(( y() Vai
(2())
(9b)
1 1-a; aj 1-a;
Eywssyo Po 1 1-a;
1-a à;
Carlo Approximation
Aspect
Model-Based Diffusion (MBD)
Model-Free Diffusion (MFD)
Target Distribution
Known (Eq. 2),, hut hard to sample
Unknown, but have data
Objective
Sample ( from high-likelihood region of po(-)
Sample y()) -o(-)
Score Approximation
Estimated using the model (Eq. (9a). Can be aug-
Learned from data
mented with demonstrations (Eqs. T and (13) Perform Monte Carlo score ascent (Fq. ( to
Backward Process
Run reverse SDE to preserve sam- ple diversity
move samples towards most-likely states
Table 1: Comparison of Model-Basee Diffusion (MBD) and Model-Free Diffusion (MFD)
Comparison between MFD and MBD. Table | highlights the key differences between MBD and MFD, which originate from two assumptions made in MBD: (a) known target distribution (o((()) given the model; (b) the objective of sampling (() from the high-likelihood region of Po(P (o) to minimize the cost function. For (a). MBD leverages po to estimate the score follow- ing Eq. Pap.. whereas MFD learns that from the data. For (b), MBD runs Monte Carlo score ascent in Eq. (6) to quickly move the samples to the high-density region as depicted in Fig, ((),, while
MFD runs reverse SDE ((-1)) = fa. (y + 1-a, Vyto log p::pp: (Y()) + /1 a;Z;, where z; is Gaussian noise, maintain the sample diversity. Given low temperature X, it can be shown that 1 (Y arg m.
i.e., the function log ()((()))) is 1 (1-a.) -smooth. Therefore, choosing the step size (1 - a.) in Eq. 6 is considered optimal, as for L-smooth functions, ((+) is the step size that achieves the fastest convergence [64].
logp;((P (Y x (1-ā.)
How diffusion helps? The diffusion process plays an important role in helping Monte Carlo score ascent overcome the local minimum issue in highly non-conves optimization problems, as shown in Fig. aa). Compared with optimizing a highly non-convex objective, Monte Carlo score ascent is applied to the intermediate distribution (+)-) = po (Y )p:oo ( (0), which is made concave by convoluting Po(.) with a Gaussian distribution Pçu(.). as shown in Fig. b).. Starting from the strongly concave Gaussian distribution PN ~ 1), with scale ēy + 0, the density is easy to sample. The covariance of the sampling density E. = ( - 11)/ is large when = _N, implying that we are searching wide space for global minima. the less-noised stage, the intermediate distribution pc(-) is more peaked and closer to the target distribution po()), and ā; + produces smaller sampling covariance Eg, to perform a local search. By iteratively running gradient ascent on the smoothed distribution, MBD can effectively optimize highly non-convex objective function as presented in Fig. 2 The MBD algorithm is formally depicted in Algorithm
Connection with Sampling-based Optimization. When diffusion step is set to N = 1, MBD effectively reduces to the Cross-Entropy Method (CEM) [1 for optimization. To see this, we can plug the estimated score Eq. (9b) into the Monte Carlo score ascent Eq. (6) and set N
( ãة (0) ()(1))= (V)))) Ev(o),ea y( w(>())) a:
where w ( = Po (Y)
Ey(o)yy(1) woY()))
exp( ( and ()( NV y(u) (2_ 1)/). This precisely mirrors the update mechanism in ag
CEM, which aims to optimize the objective function (CEM(Y(()) = (O) and determine the sampling covariance ΣCEM = (1) 1) thus linking the sampling strategy of CEM with the a
schedule in MBD. The advances that distinguish MBD from CEM-like methods are (1) the careful
"See more elaborations in Appendix A.2

---

Algorithm 1 Model-based Diffusion for Generic Optimization
1: Input: y()) ~N(00,
2: for i N to 1 do
3: Sample yx(*) ~ N y(
Vā--i (a 1)1)
4: Calculate Eq. (b) 4(1) (2(0)= En(orpti r pa(1(())
Ex(0) No(Vio))
5: Estimate the score Eq.
Pap: Vyco log (ggp:(2())
y(o fom (0) (y(())
11-
I-a.
Monte Carlo score ascent Eq. 66 ((1-1) jay
(r())
ai V log (Y())
7:
end for
scheduling of and (2) the intermediate refinements on Pa, both following the forward diffusion process. This allows MBD to optimize for smoothed functions in the early stage and gradually refine the solution for the original objective. On the contrary, CEM's solution could either be biased given a large CCEM which overly smoothes the distribution as in P20.P100 of Fig. 2b), or stuck in local minima with small ECEM as in p1 of Fig. (b) where the distribution is highly non-concave.
4.2 Masel-based Diffusion for Trajectory Optimization
For TO, we have to accommodate the constraints in Eq. 4T) which change the target distribution to po(Y() x (a())))) (Y(()) (Y)))))))))))))))))))))))))) (Y(")) Given that pa( (((()) a Dirac delta function that assigns non-zero probability only to dynamically feasible trajectories, sampling from Φ; (Y(()) could result in low densities. To enhance sampling efficiency, collect batch of dynamically feasible samples ( from the distribution
() (p)()())) with model information. Proceeding from Eq. (a). and incorporating Po ()) x pa(Y (Y (p) (Y())), we show the score function is: ya Vas I y (V (pr(()))))))))))))) (p ((((())) ( l-a. ( (a (Y))) dy
Y( lo
(10a)
1- a
a-a + Va: Enoroaoo y (p
(()
(10b)
--a Eywryy () )2
y(i
+ vā p(0))
(10c)
ai Σ Eymory w(Y())) w(v(()) ps(((()) (p
Σ Ey(e (0)) w((())))))))
where (0(0)_
(10d)
The model plays crucial role in score estimation by transforming infeasible samples y( from Line in Algorithm into feasible ones y)( The conversion is achieved by putting the control part U u1: of y()) [r [ into the dynamics Eq. (1c) recursively to get the dynamically feasible samples
( (Line 4 which shares the same idea with the shooting method 22 in TO. MBD then evaluates the weight of each sample with py (Y) )p> (Y (o)) in Line 5 One common limitation of shooting methods is that they could be inefficient for long-horizon tasks due to the combinatorial explosion of
the constrained space Py (Y) x Ih= (x u,) s 0). leading to low constraint satisfaction rates.
To address this issue, we will introduce demonstration-auss -uggmenne- MBD in Section 4.3 to guide the sampling process from the state space to improve sample quality.
4.3 Model-based Diffusion with Demonstration
With the ability to leverage model information, MBD can also be seamlessly integrated with various types of data, including imperfect or partial-state demonstrations by modeling them as noisy observations of the desired trajectory p(Yaemo y()) , Given subopti- mal demonstrations, sampling from the posterior p(y(0) | Ydemo) a Pa (Y (p())) y could lead to poor solutions as the demonstration likelihood p( Yaemo model-based distribution (o (o) (y) )o ( (()) and mislead the sampling pro-
(0)) could dominate the
7

---

Algorithm 2 Model-based Diffusion for Trajectory Optimization
1: Input: y(M) ~N(00,
2: for io = N to 1 do
3: Sample yx(*) ~ VV Vá.-i 0L-1 ( -1)
4: Get dynamically fcasible samples: 2 rollout(()
5: Calculate (79)) with Eq. 10cl (model only) Eq.
13 (model demonstration) Vay (7(0)
Estimate the score Eq. : V you logp:(Y(())
T-a.
7: Monte Carlo score ascent Eq. 66 y(i-))
(r (1 σ.) Vyo loya lo(o()())))
8:
end for
cess. Rather, we assess ( using pl demoo ) employing a similar technique to interchange the distribution's 's parameter with the random variable, as demonstrated in Eq. (8. to establish Pdemo (Y) p (Ydemo ( oc (Y 0²1).
To accommodate demonstrations of varying qualities, instead of fixing target to po (Y pp(ppppp))))))))))))))))))) de
("),, ) we propose seperating the (o from Pdemo (y)() to form a new target distribution
()) oc (1 (1-1) ()))))(()))pp(( )g (Y)) 1yPsemo (Y Pu (Ydem)) (y
(11)
where yi is constant balance the model and the demonstration. Here, have introduced two extra constant terms (Ydemo) and Py (og(aeen (Yáemo) to ensure that the demonstration likelihood is properly scaled to match the model likelihood Po (Y)))) With these preparations, we propose to adaptively determine the significance the demonstration by choosing 1) as follows:
1 (a )p (Y Pp (Y(()) Páemo (M (Y pp (Payee) Py P
1 1010 Pa(Y (p (P (Y 2 Paemo ( (Y pp
(12)
(apps(yeum (Ydemo) (y
When samples have high model-likelihood po. we ignore the demonstration and sample from the model. Otherwise, we trust the demonstration. With the demonstration aaggmmetd target distribution, we modify the way to calculate (()1) in Eq. 10d) as follows to obtain the score estimate:
(()) Σym o y ((()))
Pdemo (Y(0) )p>()) (Y Py (P Vdemo) } (13)
Pa Y (Y (P)())))))))))))))))))))) (Y())),
Eyioryoo w( (Y(()) ( = max
Experimental Results
The experimental section will focus on demonstrating the capabilities of MBD in: (1) its effectiveness as a zeroth-order solver for high-dimensional, non-convex, and non-smooth trajectory optimization problems, and (2) its flexibility in utilizing dynamically infeasible data to enhance performance and regularize solutions. Our benchmark shows that MBD outperforms PPO by 59% in various control tasks with 10% computational time.
we also show that MBD significantly improves sampling efficiency by an average of 23% over leading baselines in high-ddmmensionl (up to 800d) black- box optimization testbeds 23 18 56 42 41 44
Beyond control problems, in Appendix A.3
We also apply MBD to optimize an MLP a gradient-free manner, achieving 86% accuracy within 2s for the MNIST classification task (2]. which is comparable the gradient- rrasient-based optimizer (SGD with momentum, 93% accuracy). To further extend MBD to closed-loop control, we employ receding horizon strategy to MBD Appendix A.6 to update control sequence at each timestep, further improving the performance of MBD by 9.6% in terms of reward.
network with 28K parameters in a
MBD for Planning in Contact-rich Tasks 5.1
To test the effectiveness of MBD as a trajectory optimizer for systems involving non-smooth dynamics, we run MBD on both locomotion and manipulation tasks detailed in Appendix A.5.] The locomotion
A comparison between the remoosstation-auggeen MBD and the vanilla MBD is illustrated in Fig. with detailed breakdowns in Appendix A.4
8

---

Task
CMA-ES
CEM
MPP1
RL
MBD
Hopper
1.12 0.10
0.15
0.91 0.15
1.40 0.04
1.53 0.03
Half Cheetah
0.44
0.22
0.10
1.09
=
Ant
1.18 0.52
0.15
0.45
3.26
3.80 0.35
Walker2D
0.83 0.04
1.06
0.05
1.09 0.28
2.63 0.23
Humanoid Standup
0.58 0.01
0.07
0.53 0.05
0.83 0.02
0.99 0.07
Humanoid Running
0.60 0.11
0.41
0.59 0.14
1.80 0.03
2.92 0.26
Push
0.39 = 0.07
0.25 0.09
0.13 0.09
0.63 +: 0.16
0.67 = 0.10
Table 2: Reward of different methods non-continuous tasks.
"RL requires offline training and
generate a closed-loop policy so is not an apple-to-apple baseline.
Task
CMA-ES
CEM
MPPI
MBD
Hopper
29.3s
26.5s
26.4s
17 45.63s
26.5
Half Cheetah
29.5s
26.4s
26.7s
1m.8
26.8
Ant
18.4s
16.1s
16.0s
2m46.8s
16.2s
Walker2D
37.5
34.5s
34.7s
5m 1.5s
34.6s
Humanoid Standup
20.8
17.6s
17.7s
4m 5
17.7s
Humanoid Running
30.8s
29.75
29.6s
mm.7.7
30.0s
Push T
10.40.05
10.32.0s
10.32
67 m 25.6s
10.32.80
Table 3: Computational time of different methods on non-continuous tasks.
tasks includes hopper, half-cheetah, ant, walker2d, numanoid-standuu and numanoid-running The selected manipulation task, pushT' [15). presents its challenges due to the complexity introduced by contact dynamics and the long-horizon nature of the task. These tasks are widely considered difficult due to their hybrid nature and high dimensionality.
MBD is compared with the tate-offthe--n zeroth-order optimization methods, including CMA- ES 77. CEM 11I. and MPPI [59], well as reinforcement learning (RI.) algorithms (e.g.. PPO 47 and SAC on these tasks. Please note RI. is only used for performance reference not as there no existing TO method that can solve such high-dimensional discontinuous tasks as we have shown in the experiments. Model-free RL, especially PPO/SAC, widely used in such tasks and is considered the SOTA method. The difference between RL and MBD is further discussed in Appendix A.8 and A.6 The RL implementation follows the high- segh-performacc parallelized framework from Google Brax 22 elaborated in Appendix A.5.3] For the zeroth-order optimizer, sample number with the MBD. All the experiments were conducted on a single NVIDIA RTX 4070 Ti GPU. Quantitative metrics including the average step reward and the computational time tested over 50 steps repeated for 8 seeds are reported in Tables 2 and 3 MBD substantially outperforms zeroth-order optimization methods and even outperforms RL most tasks. Specifically, for the pushTT task, MBD achieves significantly higher reward than the RI. algorithm thanks to its iterative refinement process, which effectively explores the full control space while keeping fine-grained control to precisely push the object. Compared with the computationally heavy RL algorithms, MBD only requires one-tenth of time, which is similar to other zeroth-order optimization methods. The optimization process of MBD is visualized in Fig. B where the iterative refinement process with the model plays crucial role in optimizing high-dimensional tasks.
match the iteration and
a
path
-1
W/
Wio
Wi data
Figure 4: MBD optimized trajectory with data augmentation on the (a) Humanoid Jogging and (b) Car UMaze Navigation tasks. With data augmentation, the trajectory is regularized and refined to achieve the desired objective.

---

(8)
(b)
cc)
0
Modcl-based Diffusion
i=N
Figure 3: Optimization process of MBD the (a) Humanoid Standup, (b) Push T, and (c) Humanoid Running tasks. The trajectory is iteratively refined to achieve the desired objective the high- dimensional space with model information.
5.2 Data-augmented MBD for Trajectory Optimization
We also evaluate the performance of MBD with data augmentation on the Car UMaze Navigation and Humanoid Jogging tasks to see how partial and dynamically infeasible data can help the exploration of MBD and regularize the solution by steering the diffusion process.
For Car UMaze Navigation, the map blocked by U-shaped obstacles is challenging to explore given nonlinear dynamics model. Therefore, random shooting has low chance of reaching the goal region. To sample with loosened dynamical constraints, we augment MBD with data from the RRT (38] algorithm through goal-directed exploration with simplified dynamics. Fig. 4b) shows the difference between Mata-augmentt
MBD and data-free one: the former can refine the infeasible trajectory and further improve reach the goal less time, while the latter struggles to find feasible solution. The reason is that the infeasible trajectory from RRT serves as good initialization for MBD, which can be further refined to minimize the cost function with MBD.
For Humanoid Jogging, we aim to regularize the solution for the task with multiple solutions to the desired one with partial state data. Due the infinite possibilities for humanoid jogging motion, the human motion data provide a good reference to regularize MBD to converge to more human-like and robust solution instead of an aggressive or unstable one 126 45). We use data from the CMU Mocap dataset [ from which we extract torso, thigh, and shin positions and use them as partial state reference. Fig. demonstrates more stable motion generated by tata-augmented MBD.
Conclusion and Future Work
This paper introduces Model-Based Diffusion (MBD), novel diffusion-baset trajectory optimization framework that employs a dynamics model to approximate the score function. MBD not only out- performs existing methods in terms of sample efficiency and generalization, but also provides perspective on trajectory optimization by leveraging diffusion models as powerful samplers. Future directions involve theoretically understanding its convergence, optimizing the standard Gaussian for- ward process using the model information, adapting it to online tasks with receding horizon strategies, and exploring advanced sampling and scheduling techniques further improve performance.
Acknowledgments
This work was supported by NSF Grant 2154171, NSF CAREER 2339112 and CMU CyLab Seed Funding.
References
[1] Carnegie Mellon University
CMU
Graphics
Lab
motion
capture library.
http://mocap.cs.cmu.coo
10

---

[2] The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] IEEE Journals & Magazine IEEE Xplore.
[3] Sequential Quadratic Programming. In Jorge Nocedal and Stephen J. Wright, editors, Numerical Optimization, pages 529-562. Springer, New York, NY, 2006.
[4] David H. Ackley. A Connectionis/ Machine for Genetic Hillclimbing, volume 28 of The Kluwer International Series in Engineering and Computer Science. Springer US, Boston, MA, 1987.
[5] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Conditional Generative Modeling all you need for recision-Making"" July 2023.
[6] Tara Akhound-Sadegh, Jarrid Rector-Brooks Avishek Joey Bose, Sarthak Mittal, Pablo Lemos, Cheng-Hao Liu, Marein Sendera, Siamak Ravanbakhsh, Gauthier Gidel, Yoshua Bengio, Niko- lay Malkin, and Alexander Tong. Iterated Denoising Energy Matching for Sampling from Boltzmann Densities, February 2024.
[7] Youhei Akimoto, Yuichi Nagata, Isao Ono, and Shigenobu Kobayashi. Theoretical Foundation for CMA-ES from Information Geometry Perspective. Algorithmica, 64(4))698-71) -766, December 2012.
[8] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured Denoising Diffusion Models in Discrete State-Spaces, February 2023.
[9] Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: Framework for Efficient Monte-Carlo Bayesian Optimization, December 2020.
[10] Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective on diffusion- based generative modeling, July 2023.
[11] Zdravko Botev, Dirk P. Kroese, Reuven Y. Rubinstein, and Pierre L'Ecuyer. Chapter 3 The Cross-Entropy Method for Optimization. In C. R. Rao and Venu Govindaraju, editors, Handbook of Statistics, volume 31 of Handbook of Statistics, pages 35-59. Elsevier, January 2013.
[12] João Carvalho, An T. Le, Mark Baierl, Dorothea Koert, and Jan Peters. Motion Planning Diffusson: Learning and Planning of Robot Motions with Diffusion Models. In 2023 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages
Detroit,
MI, USA, October 2023. IEEE.
[13] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. WaveGrad: Estimating Gradients for Waveform Generation, October 2020.
[14] Xiang Cheng, Niladri S. Chatterji, Peter L. Bartlett and Michael I. Jordan. Underdamped Langevin MCMC: A non-asymptotic analysis, January 2018.
[15] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, June 2023.
[16] Tzuu-Shuh Chiang, Chii-Ruey Hwang, and Shuenn Jyi Sheu. Diffusion for Global Optimization Slmathbb |R|nn $. SIAM Journal on Control and Optimization, 2573)7737-753 May 1987.
[17] Murat A. Erdogdu and Rasa Hosseinzadeh. On the Convergence of Langevin Monte Carlo: The Interplay between Tail Growth and Smoothness. Proceedings of Thirty Fourth Conference on Learning Theory, pages 1776-1822. PMLR, July 2021.
[18] David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scal- able Global Optimization via Local Bayesian Optimization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[19] Peter I. Frazier. A Tutorial on Bayesian Optimization, July 2018.

---

[20] C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax A Differentiable Physies Engine for Large Scale Rigid Body Simulation, June 2021.
[21] Saul B. Gelfand and Sanjoy K. Mitter. Recursive Stochastic Algorithms for Global Optimization in Slmathbb (R]d $. SIAM Journal on Control and Optimization, 29(5):999-101, September 1991.
[22] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[23] Nikolaus Hansen. The CMA Evolution Strategy: Tutorial, March 2023.
[24] Nikolaus Hansen, Sibylle D. Müller, and Petros Koumoutsakos. Reducing the Time Complex- ity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES). Evolutionary Computation, 11(1):1-18, March 2003.
[25] C. R. Hargraves and S. W. Paris. Direct trajectory optimization using nonlinear programming and collocation. Journal of Guidance, Control, and Dynamics, May 2012.
[26] Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning Hormaan-to-Humaane
Real-Time Whole-Body Telcoperation, March 2024.
[27] José Miguel Bernández-Lobate James Requeima, Edward O. Pyzer-Knapp. and Alán Aspuru- Guzik. Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space. In Proceedings of the 34th International Conference on Machine Learning. pages 1470-1479. PMLR, July 2017.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, December 2020.
[29] Taylor A. Howell, Brian E. Jackson, and Zachary Manchester. ALTRO: A Fast Solver for Constrained Trajectory Optimization. In 2019 JEEÉ/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7674-7679, Macau, China, November 2019. IEEE.
[30] Xunpeng Huang, Hanze Dong, Yifan Hao, Yi-An Ma, and Tong Zhang. Reverse Diffusion Monte Carlo, March 2024.
[31] Chii-Ruey Hwang. Laplace's Method Revisited: Weak Convergence of Probability Measures. The Annals of Probability, 8(6):1177-118) December 1980.
[32] Wilson Jallet, Antoine Bambade, Nicolas Mansard, and Justin Carpentier. Constrained Dif- ferential Dynamic Programming: A primal-dual augmented Lagrangian approach, October 2022.
[33] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with Diffusion for Flexible Behavior Synthesis, December 2022.
[34] Chiyu Max Jiang, Andre Cornman, Cheolho Park, Ben Sapp, Yin Zhou, and Dragomir Anguelov. MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion, June 2023.
[35] B. Kappen, V. Gomez, and M. Opper. Optimal control as a graphical model inference problem. Machine Learning, 87(2):159-115 182, May 2012.
[36] Seung-Jean Kim, K. Koh, M. Lustig, Stephen Boyd, and Dimitry Gorinevsky. An Interior-Pon Point Method for Large-Scale Regularized Least Squares. IEEE Journal of Selected Topics in Signal Processing, 1(4):606-61) 617, December 2007.
[37] Marin Kobilarov. Cross-entropy motion planning. The International Journal of Robotics Research, 3 31(7):855-871, June 2012.
[38] Steven Valle. Rapidly-exploring random trees: A new tool for path planning. Research Report 9811, 1998.
12

---

[39] Sergey Levine. Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review, May 2018.
[40] Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. AdaptDif- fuser: Diffusion Models as Adaptive Self-evolving Planners, May 2023.
[41]
Jialin Liu, Antoine Moreau, Mike Preuss, Baptiste Roziere, Jeremy Rapin, Fabien Teytaud, and Olivier Teytaud. Versatile Black-Box Optimization, April 2020.
[42] Amin Nayebi, Alexander Munteanu, and Matthias Poloczek. A Framework for Bayesian Optimization in Embedded Subspaces. In Proceedings of the 36th International Conference on Machine Learning, pages 4752 4761. PMLR, May 2019.
[43] Radford M. Neal. Annealed Importance Sampling, September 1998.
[44] Leonard Papenmeier, Luigi Nardi, and Matthias Poloczek. Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces, April 2023.
[45] Xue Bin Peng. Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control. ACM Transactions Graphics, 40(4):1-20, August 2021.
[46] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-Conditioned Imitation Learning using Score-based Diffusion Policies, June 2023.
[47] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017.
[48] Guanya Shi, Wolfgang Honig, Xichen Shi, Yisong Yue, and Soon-Jo Chung. Neural-Swarm2. Planning and Control of Heterogeneous Multirotor Swarms Using Learned Interactions. IEEE Transactions Robotics, 38(2):0063-107) 1079, April 2022.
[49] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian Optimization of Ma- chine Learning Algorithms. In Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
[50] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics, November 2015.
[51] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In International Conference on Learning Representations, October 2020.
[52] Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution, October 2020.
[53] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastie Differential Equations, February 2021.
[54] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H. Bermano. Human Motion Diffusion Model, October 2022.
[55] Mare Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1049-1056, Montreal Quebec Canada, June 2009. ACM.
[56] Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search, In Advances in Neural Information Processing Systems, volume 33, pages 19511-19522 Curran Associates, Inc., 2020.
[57] Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynam- ics.
13

---

[58] Grady Williams, Andrew Aldrich, and Evangelos A. Theodorou. Model Predictive Path Integral Control: From Theory Parallel Computation. Journal of Guidance, Control, and Dynamics, 40(2):344-357) 357, February 2017.
[59] Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg. and Evangelos A. Theodorou. Model Predictive Control: Theory and Applications to Autonomous Driving. IEEE Transactions on Robotics, 1466):1603-1622, December 2018.
[60] Wei Xiao, Tsun-Hsuan Wang. Chuang Gan, and Daniela Rus. SafeDiffuser: Safe Planning with Diffusion Probabilistic Models, May 2023.
[61] Zhutian Yang, Jiayuan Mao, Yilun Du, Jiajun Wu, Joshua B. Tenenbaum, Tomás Lozano-Pérez. and Leslie Pack Kaclbling. Compositional Diffusion-Based Continuous Constraint Solvers, September 2023.
[62] Zeji Yi, Chaoyi Pan, Guanqi He, Guannan Qu, and Guanya Shi. CoVO-MPC: Theoretical Analysis of Sampling-based MPC and Optimal Covariance Design, January 2024.
[63] Zeji Yi, Yunyue Wei, Chu Xin Cheng, Kaibo He, and Yanan Sui. Improving sample efficiency of high dimensional Bayesian optimization with MCMC, January 2024.
[64] Matthew D. Zeiler. ADADELTA: An Adaptive Learning Rate Method, December 2012.
14

---

A Appendix Supplemennal Material
A.1 Notation Table
Meaning
Symbol
state, control at time state control pair at time diffused random variable step density of diffused distribution at step density of diffused r.v. at step conditioned step samples collected from proposal distribution at setp scale down factor at step i accumulated scale down factor at step dynamic feasibility density, optimality density, constraint density
Rt, u
Vt
Y()
pc())
Pag(- y0
āq
pd()).s().pp()
A.2
Convergence of Distribution with Small
We first give the definition of the volume of the sub-level set for cost J.
Definition 1. Let F Rd R. be measurable function. Define the volume of the sub-level set for given level as:
VE(t) Saa XYYERAAR(SSS)
where x denotes the indicator function.
The volume function Vs(t) plays crucial role in linking geometric properties with probabilistic outcomes in optimization and learning algorithms. This function provides a quantitative measure that helps understand how changes in parameters like influence the distribution and concentration of probability mass.
The interplay between geometry and probability, represented by Vy(t), is crucial for evaluating the convergence and stability of algorithms. provides a significant method for utilizing the PDF the random variable Y to constrain the CDF, thereby facilitating convergence in distribution.
Proposition 2. Given the target distribution Y p(.) with P(Y) exp exp(- (-3(2) Ye RR.. where is a cost function with miny J(Y) and y = arg min J(Y), and assuming that the volume function V (t( is bounded by polynomial inequalities:
(1) s V2(1) & (o)
where Poly((!) = Enoo ch and (olyu(() = EkIo tf are polynomials with coefficients satis- fying cf 0 if and only c c The exponent term satisfies that 0g E R, and 0 ag < 0 & OM co, follows that:
lim J(Y) E. J(Y") X
The cost value J(Y) converges in probability to J(Y*) À + 0.
The condition on the polynomial bounds of Vt( is generally not restrictive. For instance, consider = Me |Y Y. I. where y* the optimal point and T)e > is any constant. In this case, V)(t) Ct#,, where C' is a constant, meets the constraint in straightforward way. This condition can be extended beyond this simple scenario, as even if has multiple modes. it can still adhere this polynomial constraint.
Proof. The convergence in distribution of towards Y* as established by analyzing the behavior of the probability density function, defined up multiplicative constant. Consider the
15

---

density Yo PY( approximating Y when À approaches zero.
P(J(Y) Sumnsn pY)),,
(14a)
Lo Summsa (Y)ddd
(14b)
Lo ('eap(-7 2 S (J()==) dYdr,
(14c)
S expp-- 2 (V,(T))
dx,
(14d)
dæ
where Eq. (14ch valid since P(Y) exp (-xx) and J(Y) represents the sufficient statisties of
the distribution. We can obtain Eq. 14d) by computing the derivative of Vz(z) based on the volume definition as shown Definition
We denote Jmin miny J(Y)= 0 and Jmax = maxy J(Y) with Jmax satisfying 0 5 Jmax S +oo. We proceed analyze Eq. (14d) by performing integration by parts as shown in Eq. (1ap..
Jun
Jun
d d[exp( (-x
vS(O)
1 exp (- vi(()dt
exp
(V)
(15a)
Jm
v0)
Jun
Jan
(-") (i(tt...
exp
exp
(15b)
Juin
Jua
To establish convergence in probability, we need to demonstrate that for any small and 8 there exists sufficiently small À > 0, such that
foexp (-2)
P(J(Y)<)) =
21- Jo exp (- dV,
(16)
where the equality is due Eq. (1d). Setting 5' 1-8 suffices to show that:
8
S exp (-t) 8 J exp (-+) dV)(t)
(17)
Assuming without loss of generality that Jmax oo, becuase dV,(t) 2 0, exp(())) 0, we have:
So exp (-t)
exp (V
(18)
Je J exp xxp())) (V) S. exp (-+) (V>(1)
This ratio as in Fq. (17 can be expanded using the integral bounds and the polynomial approximations for (h), it suffices to show that
S exp ( (- ex (V)
J.º exp (-) dV 2
(19)
By inserting Eq.
(1bb) into both the numerator and denominator the LHS of Eq. (19. we obtain
So exp( (t),
exp(- exp(-2)V(()+ 1 erp( =z)(())))
(20a)
Sc exp(- dV) -exp(- -exp(-;) Se exp(- (t)
Ji exepl exppppp))))) (+))
(20b)
expl V J. 00
Jó erpl -0- 1)) Eie df
(20c)
f. exp(-t) Elz 2 ot
16

---

To bound the expression in Eq. (0c),, we first derive the following integrals by utilizing change of variables & which simplifies the expressions:
L exp f
oh di = Akt1 f.² exp( -x)x dx,
(21a)
1 dt= = skt1 fo (xp(-z)x=* dx.
exp
(21b)
For these transformed integrals, we can observe that fo exp( (xp(==))=' dx = I(ax 1), the gamma function, which well-defined for all non-negative Okk- Given that 8' a function 6, by applying the intermediate value theorem and definition of the limit of the integral, we can choose in such way that:
ce exp(-x)x dx Jo
Cg I(ax + +1,,
1 c
where C₁ denotes the ratio of coefficients in polynomial lower and upper bounds for (t). By selecting Emax = max Eo, EM to be the maximum of all such eg. ensuring coverage for all polynomial terms up to M, establish that:
So au exp (- x cf dl
Scme exp (- cjtt dt 2 8 for all k= 0, I....,,4 M.
(22)
By ensuring that <
we can conclude:
6mss
So exp (- En=o 1 dl
2 8',
(23)
S. exp (- Ek=o ofaa dl
M
Thus, the condition specified in Eq. (16) is satisfied, validating that the distribution of Y converges in distribution to Y as approaches zero.
By adding another mild assumption regarding the landscape of J near the global optimum, we can demonstrate the convergence of the random variable Y itself, rather than the convergence of J(Y). Definition 3. We denote the minimum of the complementary set of neighborhood as:
(8)
min |
J(Y)-
Proposition 4. Given the context and conditions specified in Desinitions 7 and 3 and Proposition 2 and given that has only one golbal minimizer Y", i.e. there exist small 8* that for 8 (0.5" (6) is strictly increasing. and J (88))) < co. It follows that:
)+
lim Y 2 Y 2-0
The random variable Y converges in probability to Y A
Proof. In order to prove that lima-o o Y P. Y*, We need to prove that for any sufficient small y 0
and > 0, there exists small >> such that
P(|Y - Y' 58) 1-7
(24)
From Definition 3 and due to the strict increase of (z))
|Y 5 VY (Y E Rd J(Y) - <
(25)
where < 8 S 8*, Because if 1 8, - ( contradicts D
(V)-J(V-)
17

---

Given that limx-- J(Y) 2. J(Y*). and any sufficient small 6, 0.
P(J()) JJY)))(("")))2211
(26)
Therefore, YE > O, such that
21
(27)
And From Eq. 25
have that
P( s)) P(J())
1-7
(28)
We have that Y converges in probability to Y* jie, lima-0 Y 2. Y*.
Proposition 5. Given the context and conditions specified in Propositions and 4 and the way we define the forward process as in Eq. 33 The diffused Y, converge in density to a Gaussian distribution.
lim Y 4 NN(aYY v-a.))
A
where (+) p.).)) in Eq.
Proposition is derived by using Slutsky's theorem on Proposition 4 and offers insight into choosing the stepsize discussed in Section 4.1
A.3 Black-box Optimization with MBD
As a zeroth order optimizer, MBD is capable of addressing both trajectory optimization and broader, high-dimensional unconstrained optimization challenges. Such black-box optimization tasks universally acknowledged as difficult [7] 633. We first show superior performance of MBD within this black-box optimization context. In such settings, the Bayesian Optimization technique struggles due to the computational intensity required to develop surrogate models and identify new potential solutions [8] Alternative black-box optimization strategies 23 not limited by computational issues but tend to he less efficient because they do not estimate the black-box function as accurately. MBD's effectiveness is evaluated using well-known highly non-convex black-box optimization benchmarks: Ackley 4 and Rastrigin [9], each tested across three different dimensionalities. Com- parisons were made with CMA-ES 23 TuRBO 18] LA-MCTS (56), HesBO (42, Shiwa [1 and
BAXUS
Acklby-S00d
-8-
-75
300 0381
1001
1050
3000
3381
3000
1000
3100
Raztrigin 200d
Basitign: 400d
Bastrighn 8003
1000
5000
1000
-3000
11000
2000
-4000
-2000
-1:000
3007
1380
1000
000
1000
3000
5000
5000
2000
1000
1000
2000
3000
4100
5000
000
TURBO
LA..CTS
1l-00
CMA-FS
Shies
BAXUS
3/00
Figure 5: Performance of MBD on high-dimensional black-bor optimization benchmarks. MBD outperforms other Gaussian poncess-based Bayesian Optimization methods by a clear margin.
Fig- showw the performance of MBD on the Ackley and Rastrigin benchmarks. MBD demonstrates superior performance over other algorithms for several reasons. Firstly, the implementation of
18

---

scheduled forward process that determines the total number of samples consequently boosts sample efficiency. Secondly, the application of Monte Carlo score ascent on various p: (Y()) facilitates its escape from local optima of varying scales. It is important computational efficiency may not be entirely fair, given that black-box optimization problems typically involve functions that are costly to evaluate. However, MBD markedly outperforms other Gaussian Process-based Bayesian Optimization approaches, achieving computational time savings of more than twentyfold, similar to the improvements observed with different evolutionary optimization strategies.
acknowledge that comparing
Here are the implementation detail for the benchmarks. For the BO benchmarks, the experiments were conducted on an A100 GPU because of the high computational demands of the Gaussian Process Regression Model incorporates.
TuRBO: TuRBO implemented based on tutorials from Botorch
LA-MCTS: LA-MCTS, we refer to authors' reference implementations, and use TuRBO as its local BO solver [56].
HesBO:
For HesBO, we refer to authors' reference implementations 42 We transformed default
GP component into Gpytorch version for faster inference speed on GPU. We set the embedding dimension to 20 for all tasks
CMA-ES: We use pycm to implement CMA-ES, and use default setting except setting population size eqauls to batch size.
Shiwa: We use Nevergrad to implement Shiwa, and use default setting to run experiments.
BAxUS: We refer to the authors' reference implementations 4
A.3.1 MBD for DNN Training without Gradient Information
To further demonstrate the effectiveness of MBD in high-dimensional systems, we apply MBD to optimize MIP network for MNIST classification 2 without access the gradient information. MBD achieve 85.5% accuracy with 256 samples within 2s, which is comparable to the performance of the SGD optimizer with momentum 99.7%% accuracy). We use MLP with hidden layers, each with 32 neurons, and ReLU activation function. The input is flattened to 784 dimensions, and the output a 10-dimensional vector. We use cross-entropy loss as the objective function. The network has 27, 562 parameters in total, which makes sampling-bass based optimization challenging. MBD can effectively optimize the network with a small number of samples, demonstrating its effectiveness in high-dimensional black-box optimization tasks.
A.4 MBD with Demonstration Explaination
Data angteeugmeeteed MBD calculate the score function with demostration as follows:
((-1)) = Vas
1 (y(1) + +11 (1-ā:) (ogp:(
(29)
Vyc log y()
+ Va ((0)
(30)
1- az where (())_ Eyoray ((0)((
(31)
Eyoor (((()))
Wmodel (model(y(0)) (y(0)) = (y)( (p)()))))))))))))))))))))))))))) (x
w(( w(Y()))= max
(32)
Waemo (( (0))) (00))===> Pdemo ((0))p, Yeeeo ) Veewo)
19

---

Objective and Density
Backward Process with MC Score Ascend
0.0
0.5
Objective
0.02
Demstty
-0.1
-02
-0.1
-1.0
-100
00.75
-0.00
-025
0,00
0.25
10.00
0.755
1.00
10
30
40
50
Diffusion Step
200
Intoastibee Region
Data Polut
Disicsl
w/ data
datt
Thoa Point
Figure 6: MBD with data without data nonconvex function with constraints || -044/ > 0.3. We want MBD converge the optimal point * with the help of demonstration data Although the demostration point is not optimal, MBD can still converge to the optimal point with the guidance of the demonstration data. Here data serves as a regularization term to guide the diffusion process to the negative optimal point while allowing to use model further to refine the solution.
where demonstrate likelihood term Wdemo ( (Y()) will draw samples towards data without considering the model. Given w Wdemo: Y (0) Ey(0)y(() Y Paam (Y())
Eycmey( Pieao (V()) Ydemo- The score function would be
Vyc logp:(Y()))
I-a, I-a. Ydemo, which means the score function linear combination of
(t)
the current sample and the demonstration data.
If we don't 't use Eq. 11 and employ the posterior distribution p(Y()))(())))))) (Y))mo) oc Po (Y (0) Pdemo (Y())),
will yields update weights w WWaemeWameell which will draw samples both model and demonstra- tion data. If the demonstration data is not optimal, the final solution will be compromise between the model and demonstration data. In Fig. 6 the resulted solution will lie between optimal point * with the help of demonstration data
Using the max function in w can aviod this issue. In the early stage while (Y) is low due to poor sample quality, Wdeeno will dominate thanks to the high p.r (Ydemo). This will draw samples towards the demonstration data as shown in the earlier stage of Fig. [ As the sample quality improves and P. ())) Pr(Ydemo), Wodel will dominate and the sample will converge the optimal point.
A.5
Experiment Details
A.5.1 Simulator and Environment
We leverage the GPU-accelerated simulator Google Brax (20 to design the locomotion and manipula- tion tasks. All task is set use positional backend in Brax except for the pushT task, which uses the generalizable backend for better contact dynamics simulation. Here we provide a brief description of each task implementations:
1. Ant: The Ant task is a 3D locomotion task where the agent is required to move forward as fast as possible. The reward composed of the forward velocity of the agent and control cost, same as the original Brax implementation. The control dimension is 8.
2. Hopper: The Hopper task is 2D locomotion task where the agent is required to jumping forward as fast as possible We use the same reward function as the original Brax implemen- tation. We modify the simulation substeps from 10 to 20 for longer planning horizon given the same control node. The control dimension is 3.
3. Walker2d: The Walker2d task is a 2D) locomotion task where the agent required to walk forward. The reward is composed of keep the agent upright and moving forward. The control dimension is 6.
4. Halfcheetah: The Halfcheetah task is a 2D locomotion task where the agent required run forward. The reward is composed of the forward velocity of the agent and control
20

---

cost. We follow the same reward function as the original Brax implementation. The control dimension
5. Humanoidrun: The Humanoidrun task is a 3D locomotion task where the agent is required run forward. The reward is composed of the forward velocity of the agent and standing upright. Here we also modify the simulation substeps from 10 to 20 for longer planning horizon. The control dimension is 17.
6. Humanoidstandup: The Humanoidstandup task is a 3D locomotion task where the agent is required to stand up. The reward is the upright torso position of the agent. The control dimension is 17.
7. PushT:
The PushT task is 21) manipulation task where you apply force sphere to push the T-shaped object to the target location. The reward composed of the distance between the target and the object and orientation difference between the target and the object. To make the task more challenging, we randomize the target location 20cm away from the initial position and make sure the rotational angle is greater than 135 degrees, which makes it hard to solve the task with single continous contact policy. The control dimension is 2.
8. Car2D: We implement 2D task with standard bicycle dynamics model, where state is x = [x, y, 0, U, 8]. and action is u = [a, 6]. The dynamics is defined as 4 I' f(x u) [ (), ((), E tan()), a, 6. The constraints
in the middle of the map, where the car cannot enter. The reward is composed of the distance between the target and the car and the control cost. The control dimension 2.
defined as the U-shape
A.5.2 MBD Hyperparameters
In general, MBD is very little hyperparameters to tune compared with RL. We use the same hyperpa- rameters for all the tasks, with small tweaks for harder tasks.
Task Name
Horizon
Sample Number
Temperature À
Ant
50
100
0.1
Halfcheetah
50
100
0.4
Hopper
50
100
0.1
Humanoidstandup
50
100
0.1
Humanoidrun
50
300
0.1
Walker2d
50
100
0.1
PushT
40
200
0.2
Table 4: MBD hyperparameters for various tasks
For diffusion noise schedulling, we use simple linear scheduling 30 1 10 and (N = 1 10 and the diffusion step number 100 across all tasks. Each step's a; calculated as aq 3i.
A.5.3 Baseline Algorithms Implementation
For reinforcement learning implementation, we strictly follow the hyperparameters and implementa- tion details provided by the original Brax repository, which optimize for the best performance. For our self-implemented PushT task, the hyperparameters is ported from Pusher task in Brax for fair comparison. The hyperparameters for the RL tasks are shown in Table and Table
For the zeroth order optimization tasks, we the same hyperparameters as the MBD algorithm.
A.5,4 Demonstration Collections
For RRT algorithm in Car2D task, set the max step size 0.2, and the max iterations to 1000 given the maximum episode length is 50.
For the demonstration collection in Humanoid Jogging task, we first download the mocap data which contains each joints' position the world frame. Then we use the joint data to calculate the position of torso, thigh and shin position partial state reference for our task.
21

---

Environment
Algorithm
Timesteps
Reward Scaling
Episode Length
Ant
PPO
100M
10
1000
Hopper
SAC
6,55M
30
1000
Walker2d
PPO
50M
1000
Halfcheetah
PPO
50M
1000
Pusher
PPO
50M
1000
PushT
PPO
100M
1.0
100
Humanoidrun
PPO
100M
0.1
100
Humanoidstandup
PPO
100M
0.1
1000
Table 5: General RL configuration for various environments
Environment
Minibatches
Updates/Batch
Discounting
Learning Rate
Ant
32
4
0.97
3 10-4
Hopper
32
4
0.997
6 10
Walker2d
32
8
0.95
3 x 10
Halfcheetah
32
8
0,95
3 10-4
Pusher
16
8
0.95
3 10
PushT
16
8
0,99
3 10
Humanoidrun
32
8
0.97
3 10
Humanoidstandup
32
8
0.97
6 10
Table 6: RL specifics for various environments
A.6 MBD for Online Control
Even though MBD is designed as a trajectory optimization algorithm, it can be naturally extended receding horizon control as shown in Algorithm 3 By conditioning the planning each step's observation, MBD's MPC extension can further improve the performance of MBD by 9.6% in terms of reward, especially given control noise shown in Fig. 7
The online rumning frequence of MBD MPC is shown in Table on RTX 4070Ti GPU. Please note that the frequency calculated under the assumption of solving the whole 50 steps TO problem without reduced model at each iteration. Besides, the Brax code environment we used is not optimized for GPU, the actual frequency could be higher with optimized environment. In our work we just use Brax as a simple and easy-to-use option. As the major computation time of MBD is spent on the forward dynamics simulation, it can be further improved by using more efficient physics engine.
Environment
Frequency (Hz)
Hopper 4.56
HalfCheetah
4.51
Ant
10.28
Walker2d
3,49
Humanoidstandup
6.82
Humanoidrun
4.03
Table Online Running Frequency of receding horizon MBD
A.7 Sample Number Abalation
Given that sampling-based optimization is the core of MBD, we ablate the sample number in Fig. g can see that MBD converges to the optimal point with few 128 samples. The harder the task is. the larger performance gap between MBD and other TO solvers.
22

---

Algorithm 3 Model-based Diffusion with Receding Horizon
1: Initialize: Optimize trajectory Jo:T; NoT-1 with MBD
2: for {= 0 to Cini do
Observe the state Zi
4: Optimize trajectory Tuuts Mut-T-1 with single-step MBD
5:
Apply the first control input Me to the system
6:
Shift trajectory for next initialization: It-1i++, M++1:+T-11
7: end for
Algoricha
RL
(nooi))))
MBD
(noiss))
MBD MBD( (receding)
(nolaay
Roward
Hopper
HallChentah
Ant
Walker2D
Humamnid
Humanoid Running
PushT
Standup
Enviramment
Figure 7:
performance between RL, MBD and MBD receding horizon version with both perfect model and noisy model (with 5% control noise). Given perfect model, we find MBD's performance can be further augmented by 9.6% with naive receding horizon planning and leads RL by larger 74.2%. Given noisy model, MBD receding horizon version still outperforms RL by 65.3%.
Comparison
A.8 Objective Function Abalation
Given that RL and TO have different objective settings. especially the horizon difference, conducted an ablation study by swapping the optimization objectives of MBD and RI. RI optimizes for longer horizon discounted reward J Σ/n 7rre, H 1000, 7 < 1 while MBD optimizes for shorter horizon undiscounted cumulative rewards = E/iyaD H 50, y= 1. Figure 9 shows the performance of MBD and RL under each other's optimization objectives.
MBD outperforms RL by 44.5% under the RL objective and 805.5% under the TO objective. The results demonstrate that MBD's superior performance is attributed to its better diffusion-style iterative optimization process compared with RL's random exploration.
23

---

Ant
Hopper
WallerDD
HallCheetala
1,5
Tewrat
1.0
1.0
0.5
0.3
0.0
66
128
256
512
10212045
256
312
10242018
125
236
512
10242948
04
122
2065
502
1024
2045
HumarokdRaaminng
HommoonidSStnnuup
PushT
0.6
0.5
0.6
0.8
Bevard
Reword
Beword
0.6
0.4
0.4
0.2
-0.2
0.0
0.00
N
CMA-ES
MPPT
MBD
Figure 8: Ablation study the number of samples's effect the performance of MBD and other TO solver. For lower-dimensional task like Ant, Hopper, MBD's performance less sensitive to the number of samples. For higher-dimensionall task like Humanoid, MBD's performance is more sensitive to the number samples but still outperforms all the baselines.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section Iiitt the main contributions of the paper, which are accurately reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
be expected to generalize other settings.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section discusses the limitations of the proposed approach.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors

---

Algorithm RL(y = 1, H = 50)
RL(y < 1, = 1000)
MBD(y H, = 50)
MBD(y 1,H, 1000)
Reward
Hopper
HalfCheetah
Ant.
Walker2D
Humanoid Standup
Humanoid Rumning
PushTT
Environment
Figure 9: Ablation study RL and TO objective function. y= 1, 50 is TO objective, Y < 1, H = 1000 is RL objective, where 7 is the discount factor and H the horizon. For both objective function, MBD outperforms RL by 44.5% and 805.5% average. We also find RL's overall performance better given the discounted objective while MBD's performance is better given the total objective, which is consistent with our main results.
should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only lested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- lant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: This paper includes theoretical results, and all the assumptions and proofs are provided the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
25

---

All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix A.5 provides all the necessary information to reproduce the main experimental results. The code and data are also provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
26

---

The paper provides open access to the data and code, with dedicated README files and instructions reproduce the main experimental results comes with the supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis. cc/
Sublic/guides /ttd//deoo//oeedeiiiieeee CoodeSumissiiooooll1
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https
ctips.cc/public. htwpi//gguiee//ee http:////guuueee//ooooobbaaaaaaa httpsss/ooooeSuuussssssoooooeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix A.5 provides all the necessary information understand the experi- mental results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Main quantitative results in Table report the mean and standard deviation over 8 runs given different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
27

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the hardware requirements in Appendix A.5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Ethics.
The research conducted in the paper conforms with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper does not address societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We mentioned the package used in the paper Appendix A.5
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "TdskRx0DzF", "reviewer": "Reviewer_sfi9", "summary": "The paper presents Model-based Diffusion (MBD), a framework to train diffusion models to solve trajectory optimization (TO)  problems. In particular, the method assumes that the cost function associated with a TO problem can be evaluated for any trajectory of decision variables (states and control variables), which it then leverages to compute the target distribution of optimal solutions ($p_0(Y^{(0)})$) up to a normalizing constant. The denoising process in MBD performs a gradient ascent on intermediate distributions which are smoothed, easier to optimize versions of $p_0(Y^{(0)})$.\n\nThe method deals with unconstrained and constrained TO problems and is also able to incorporate demonstrations. MBD outperforms sota RL and also  sampling-based TO methods  in contact-rich tasks.", "questions": "- What are the challenges of adapting your method to online tasks with receding horizon strategies?  Is it a matter of training MBD to solve TO problems starting from more initial states? \n\n-Once MDB has been trained, how long does it take to run the denoising process to get a new trajectory?", "limitations": "Limitations of the method are mentioned in the paper as potential lines of future work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is evaluated on contact-rich, high-dimensional tasks and outperforms the baselines in terms of reward while keeping a computational time similar to the fastest baselines. \n\n- The paper is well-written and the main ideas are clear and well-motivated. Furthermore, the method shows interesting connections to multi-stage optimization and also to the Cross Entropy Method for sample-based optimization.\n\n- The method does not have many hyperparameters and they are robust across most tasks.", "weaknesses": "- While the comparison against RL is valuable and gives insights into the overall performance of MDB, the paper should better emphasize that the rewards reported in Table 2 are obtained by rolling out RL policies in an online closed-loop fashion, while the reward associated with MBD is computed according to an open loop execution.  Given that the paper lists the adaptation of MDB to online setup as future work, and that the rewards are computed in different ways, simply stating that MDB outperforms PPO by 59% can be misleading.  Qualifying such statements with an explanation of the different ways that rewards are computed would further improve the quality of the submission. \n\n[Minor] \nTypo L213 esitimation -> estimation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the comments and suggestions. We address the comments and suggestions as follows:\n\n---\n\n> the paper should better emphasize that the rewards reported in Table 2 are obtained by rolling out RL policies in an online closed-loop fashion. \n\nThanks for the suggestion. \nWe will further clearify this difference in updated maunscript.\nFor the reason why we use RL as baseline, please refer to general response 1 and 2 for the detailed explanation of comparing MBD with RL and extend it to receding horizon version. \nOverall, MBD still outperforms RL in long horizon setting and can be easily extended to online setting with receding horizon strategy.\n\n---\n\n> What are the challenges of adapting your method to online tasks with receding horizon strategies? Is it a matter of training MBD to solve TO problems starting from more initial states?\n\nPlease refer to general response 2 for further details about the extension of MBD to online setting. \nIn short, there is no fundamental limitation to extending MBD to closed-loop control with a receding horizon thanks to our fully model-based nature.\n\n---\n\n> Once MDB has been trained, how long does it take to run the denoising process to get a new trajectory?\n\nPlease refer to general response 3 for the training process clearification.\nIn short, MBD does not require training components since we can calculate the score explicitly with model but can be further improved with extra data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OMS6gdDtd9", "reviewer": "Reviewer_X3LL", "summary": "This paper presents Model Based Diffusion (MBD), a novel approach to trajectory optimization that leverages a diffusion process. Unlike traditional diffusion models where denoising networks are trained on data using a score matching loss, MBD directly computes the score function for a given target distribution. Specifically, for a trajectory optimization problem, the unnormalized target distribution is formulated as a product of three terms: an optimality term that depends on the cost function, a dynamical feasibility term, and a constraint satisfaction term. The authors demonstrate that in each iteration of the reverse diffusion process, the score function can be approximated using Monte-Carlo estimation – a weighted sum of samples drawn from a Gaussian distribution parameterized by the current trajectory, where the weight of each sample is the target density evaluated at that sample. Additionally, they show how their approach can be extended to handle scenarios with noisy observations of trajectories from the target distribution. Finally, they present experimental results on locomotion and manipulation tasks, demonstrating that their method outperforms other trajectory refinement techniques and RL algorithms in both average step reward and computational time.", "questions": "- The paper primarily evaluates MBD on tasks with relatively short horizons (e.g., 50 steps for most locomotion tasks). It would be valuable to understand how the performance of MBD scales with increasing horizon lengths. Do the computational benefits over RL persist for longer horizons? \n- How does the performance of MBD vary as a function of the number of samples used in the Monte Carlo approximation?\n- For the RL agents, it looks like the full episode length (e.g. 1000 steps for locomotion tasks like hopper, halfcheetah) is used. The RL agents are presumably trained to maximize rewards over the entire episode, while is MBD optimized for a much shorter horizon. Is this then a fair comparison? The paper could benefit from a discussion of this comparison or additional experiments with RL agents trained on shorter horizons.\n- How do you envision incorporating conditioning signals in MBD? This would be important for adapting MBD to a receding horizon strategy, where the trajectory is re-planned at each time step based on new observations.", "limitations": "The authors have not discussed the limitations of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper introduces an innovative approach that leverages diffusion models directly as solvers for trajectory optimization. The key advantage is that if the cost function, dynamics, and constraints can be specified for a given problem, the proposed iterative refinement approach can obtain samples from the desired distribution without having to rely on any demonstration data. If demonstration data is available, the authors demonstrate how it could be used to augment their approach.\nThe paper is also well-presented and easy to understand.", "weaknesses": "The requirement for explicit specification of the cost function, dynamics, and constraints can be a limitation in real-world applicatioms. It would also be interesting to see how MBD would perform in the sparse reward or goal completion settings.\nThe performance of MBD presumably heavily relies on the accuracy of the provided dynamics model. Inaccurate models could lead to suboptimal trajectories and compounding errors. It would be useful to study how robust MBD is to varying degrees of model inaccuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the comments and suggestions. We address the comments and suggestions as follows:\n\n---\n\n> The requirement for explicit specification of the cost function, dynamics, and constraints can be a limitation in real-world applications.\n\nThe requirement of dynamics and cost is defined by the nature of the TO problem we studied in the paper, i.e. given a model and cost function, how to solve the optimal control problem efficiently.\nThe usage of dynamic model and cost function actually could be one key advantage of MBD compared with MFD.\nFor in most robotics tasks including manipulator control, locomotion, and navigation, we have access to the first principle model of the system and the cost function is usually designed by the user.\nOur method makes full use of the model and cost function to enable a general and efficient trajectory optimization algorithm without training from demonstration data for each platform.\n\nBut we totally agree with the reviewer that in some cases, accurate model and objective function could be hard to obtain, which can be further addressed by model learning methods as we mentioned in general response 3.\n\n---\n\n> It would also be interesting to see how MBD would perform in the sparse reward or goal completion settings.\n\nActually, our `pushT` and `Car UMaze Navigation` tasks are exactly the sparse reward or goal completion settings.\nFor `Car UMaze Navigation` task, the agent only gets reward when it reaches the goal, which make standard sampling-based algorithm like MPPI fails due to the exploration difficulty.\nSame for the `pushT` task, the agent only gets reward when pushing the object overlapping with the random goal position and even large scale RL algorithm like PPO cannot solve it due to the sparsity reward and goal completion settings.\n`pushT` task are used to be solved by complex convex decomposition method [1] or imitation learning method [2].\nMBD is the first sampling-based algorithm that can solve these tasks efficiently.\n\n**The superior performance of MBD in sparse-reward tasks comes from the forward noise injection process, which make the sparse reward spread to larger space and make the goal completion more reachable**. \nIn the early diffusion step, larger noise is convoluted into the objective function, which helps to makes it easier for MBD to find the space where the goal is reachable.\nIn the later smaller noise stage, MBD can further refine the trajectory to reach the goal. \n\n[1] B. P. Graesdal et al., “Towards Tight Convex Relaxations for Contact-Rich Manipulation,” Jul. 05, 2024, arXiv: arXiv:2402.10312\n\n[2] C. Chi et al., “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion,” Jun. 01, 2023, arXiv: arXiv:2303.04137\n\n---\n\n> Inaccurate models could lead to suboptimal trajectories and compounding errors. It would be useful to study how robust MBD is to varying degrees of model inaccuracy.\n\nPlease refer to general response 4 for the robustness of MBD to noisy dynamics.\nWe also tested MBD's performance under weight-sensitive task like humanoidstandup. We plan/train the trajectory with nominal model while test it with a model with 20% mass perturbation as shown in Table 4.\nThis again highlights the robustness of MBD thanks to the iterative refinement process in the optimization, which make the solution less sensitive to the model inaccuracy.\n\nTable 4: Mean Reward of HumanoidStandup\n| Method | RL | MBD | MBD (Receding Horizon) |\n|-|-|-|-|\n| Nominal Model | 0.83 | 0.99 | 1.05 |\n| 20% Mass Perturbation | 0.72 | 0.84 | 0.89 |\n\n\nLastly, we want to emphasize that MBD is a TO solver that leverages the model information, which means MBD can adapt quickly to the model inaccuracy by updating the trajectory with the new model. \nThis is a key advantage of MBD compared with MFD, which requires recollecting data and retraining the model given new dynamics. \n\n---\n\n> The paper primarily evaluates MBD on tasks with relatively short horizons. It would be valuable to understand how the performance of MBD scales with increasing horizon lengths. Do the computational benefits over RL persist for longer horizons?\n\nPlease refer to general response 1 for optimizing the RL objective with MBD. MBD still outperform RL by $44.5\\%$ when horizon length is increased to 1000 steps.\n\n---\n\n> How does the performance of MBD vary as a function of the number of samples used in the Monte Carlo approximation?\n\nThanks for the suggestion and we will include the ablation study of the number of samples in the Monte Carlo approximation in the revised version. Reviewer can refer to Figure 3 in the accompanying PDF. \nThe results indicated that MBD is less sensitive to the number of samples compared with other baselines and can handle most tasks well even with 128 samples.\n\n---\n\n> The RL agents are presumably trained to maximize rewards over the entire episode, while is MBD optimized for a much shorter horizon. Is this then a fair comparison?\n\nWe thank the reviewer for the insightful question. Please refer to general response 1 for the detailed explanation of comparing MBD with RL by swaping the optimization objective. \nIn short, the comparision is fair due to RL objective has a discount factor. MBD can outperform RL in both short and long horizon settings.\n\n---\n\n> How do you envision incorporating conditioning signals in MBD? This would be important for adapting MBD to a receding horizon strategy, where the trajectory is re-planned at each time step based on new observations.\n\nMBD can be easily extended into the online setting by adopting a simple receding horizon strategy in general MPC algorithms. \nPlease refer to general response 2 for further details about the extension of MBD to online setting.\nIn short, by combing with the most naive receding horizon strategy, MBD can finetune the original plan at each step easily without modification to the core algorithm.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "njnwaGfcb0", "reviewer": "Reviewer_YGTC", "summary": "This work introduces model-based diffusion (MBD) to solve trajectory optimization (TO) problems without relying on data. The key idea is to explicitly compute the score function by leveraging the model information in TO problems. This paper also shows that MBD has interesting connections to sampling-based optimization. The empirical evaluations demonstrate that MBD outperforms state-of-the-art TO methods.", "questions": "Please refer to the Weaknesses.", "limitations": "Limitations are not discussed in the manuscript.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The model-based diffusion framework enables an effective trajectory planner.\n- This algorithm shows good performance on various tasks compared to other state-of-the-art algorithms.", "weaknesses": "- The theoretical derivation and notation in the methodology are quite confusing and hard to follow.\n- The comparison to some simple baseline of RL, such as PPO, cannot demonstrate the real advantages.  \n- In line 137 and equation $(3)$, there may be an incorrect formula: \n$$\np_{i|i-1}(\\cdot, Y^{(i-1)}) \\sim \\mathcal{N}(\\sqrt{\\alpha_i}Y^{(i-1)},1-\\alpha_iI)\n$$\nand \n$$\np_{i|0}(\\cdot, Y^{(0)}) \\sim \\mathcal{N}(\\sqrt{\\overline{\\alpha}_i}Y^{(0)},1-\\overline{\\alpha}_iI)\n$$\n\n- In equation $(8)$, there may be an incorrect formula:\n$$\n\\phi_i(Y^{(0)})\\propto\\cdots\\propto\\mathcal{N}(Y^{(0)}-\\frac{Y^{(i)}}{\\sqrt{\\overline{\\alpha}_i}},\\frac{I}{\\overline{\\alpha}_i}-I)\n$$\n\n- In all algorithms, it is evident that the calculation step can be simplified: \n\n$Y^{(i-1)} \\leftarrow \\frac{1}{\\sqrt{\\alpha_i}}(Y^{(i)}-Y^{(i)}+\\sqrt{\\overline{\\alpha}_i}\\overline{Y}^{(0)}(\\mathcal{Y}^{(i)})) = \\frac{\\sqrt{\\overline{\\alpha}_i}}{\\sqrt{\\alpha_i}}\\overline{Y}^{(0)}(\\mathcal{Y}^{(i)})=\\sqrt{\\overline{\\alpha}_{i-1}}\\overline{Y}^{(0)}(\\mathcal{Y}^{(i)})$\n\nNote that this coefficient $\\sqrt{\\overline{\\alpha}_{i-1}}$ is canceled at the first step. Also, note that in equations $(9b)$, $(10d)$, and $(13)$, it is a weighted average of the samples from $\\mathcal{Y}^{(i)}$. It may be clearer if the calculation steps are more simplified and easy to understand.\n\n- In Table 3, the computational times of the CEM and MC-MBD algorithms are close. I'm wondering if this table includes the computation time for environment interaction. It might be fairer to compare just the computation times for the algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the comments and suggestions. We address the comments and suggestions as follows:\n\n---\n\n> The theoretical derivation and notation in the methodology are quite confusing and hard to follow.\n\nOur approach utilizes Reverse SDE, a technique commonly used in diffusion models, to optimize trajectories under an annealed schedule. Similar to Model Free Diffusion, estimating the score function is crucial. Instead of learning from large data sets, we use a novel sampling strategy. We employ importance sampling with a Gaussian proposal distribution centered on the current $Y^{(i)}$, which improves the use of model information and enhances our model's efficiency and accuracy in dynamic settings.\n\nHere we provide some more details on the purpose of each step in the derivation. Eq. (7) is applying the gradient rule to the diffused distribution; from (7) to (9), the goal is to try to rewrite the integral in (7) as an expectation on $Y^{(0)}$, for which purpose we need to rewrite the density function $p_{i|0}$ as a samplable proposal density function for $Y^{(0)}$ (which is (8)). Lastly, in eq. (9) we replace the expectation with a Monte-Carlo sampling of $Y^{(0)}$. We also provicde a more detailed Notation Table below. We hope these helps and we would appreciate if you can provide more specific suggestions on improving the presentation.\n\n\nTable5: Notation Table\n| Meaning | Symbol |\n|--|--|\n| state, control at time t  | $x_t,u_t$|\n| state control pair at time t | $y_t$ |\n| diffused random variable at step i | $Y^{(i)}$ |\n| density of diffused distribution at step i | $p_i(\\cdot)$ |\n| density of diffused r.v. at step i conditioned on step j| $p_{i\\mid j}(\\cdot \\mid \\cdot)$ |\n| samples collected from proposal distribution at setp i | $\\mathcal{Y}^{(i)}$ |\n| scale down factor at step i | $\\alpha_i$\n| accumulated scale down factor at step i | $\\bar{\\alpha}_i$ |\n| dynamic feasibility density, optimality density, constraint density | $p_d(\\cdot), p_J(\\cdot), p_g(\\cdot)$ |\n\n\n---\n\n> The comparison to some simple baseline of RL, such as PPO, cannot demonstrate the real advantages. \n\nFor the reason why use model-free RL as one performance reference, please refer to general response 1. \nGenerally speaking, due to the complexity of the task and non-smoothness of the dynamics, classic gradient-based optimization algorithms like CEM and CMA-ES failed to solve the task.\nFor the reference purpose, we include model-free RL algorithms like PPO and SAC, which are generally considered to be state-of-the-art in such tasks [2,3], into the comparison.\nOur implementation of PPO and SAC are adopted on the Google Deepmind Brax [1], which involves common improvements like large-scale parallelism, reward scaling, hyperparameter sweeps and state normalization, etc.\nSo we believe that the our well-tuned implementation of PPO and SAC can provide a strong reference to demonstrate the effectiveness of our proposed method.\n\n[1] C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem, “Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation,” Jun. 24, 2021, arXiv: arXiv:2106.13281.\n\n[2] S. Çalışır and M. K. Pehlivanoğlu, “Model-Free Reinforcement Learning Algorithms: A Survey,” in 2019 27th Signal Processing and Communications Applications Conference (SIU), Apr. 2019, pp. 1–4. doi: 10.1109/SIU.2019.8806389.\n\n[3] F. E. Dorner, “Measuring Progress in Deep Reinforcement Learning Sample Efficiency,” Feb. 09, 2021, arXiv: arXiv:2102.04881.\n\n---\n\n> In line 137 and equation $(3)$, there may be an incorrect formula. In equation $(8)$, there may be an incorrect formula\n\nYes, we made a typo in the equation $(3)$ and $(8)$ by replacing variance with standard deviation. We will correct the typos in the revised manuscript. Please note that this does not affect other derivations in the paper and all the subsequent equations are correct. Thank you for pointing out the errors. \n\n---\n\n> Note that this coefficient $\\sqrt{\\overline{\\alpha}_{i-1}}$ is canceled at the first step. Also, note that in equations $(9b)$, $(10d)$, and $(13)$, it is a weighted average of the samples from $\\mathcal{Y}^{(i)}$. It may be clearer if the calculation steps are more simplified and easy to understand.\n\nThat's a quite insightful observation. \nWe would like to point out that **there are two perspectives to interpret the calculation steps**. \nFor the first perspective, we first calculate the score function and then plug it into the reverse process, which is the standard way to derive the backward process in the diffusion model.\nFor the second perspective, as we mentioned in Sec 4.1 *Connection with Sampling-based Optimization*, plugging in the score actually leads to an interesting connection with existing optimization algorithms in single-step diffusion case, which coincides with the reveiwer's opinion here.\n\nWe keep the score expression in the main text for the sake of completeness, but we will provide a alternative version in the appendix. \nThank you for the suggestion!\n\n---\n\n> In Table 3, the computational times of the CEM and MC-MBD algorithms are close. I'm wondering if this table includes the computation time for environment interaction. It might be fairer to compare just the computation times for the algorithms.\n\nYes, the computational times in Table 3 include the computation time for environment interaction. The reason why their computational time is close is that we match the iteration number of baseline algorithms to MBD diffusion steps for fair comparison, which is a common practice for the planning algorithm used in MBRL. We will clarify this in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OZIQYFP6RB", "reviewer": "Reviewer_p88D", "summary": "The paper proposed MBO, a novel method to solve trajectory optimization problems with diffusion models. MBD formulates the optimization problem as a sampling problem and estimates the score function using the dynamic model.", "questions": "There are some minor questions for better understanding.\n\n- For model-free diffusion, we need to train the score function estimator through the data. In model-based diffusion, which part should be parametrized by the neural network and trained with the data?\n\n- MBD is compared with the baselines, including zeroth-order optimization and RL algorithms. Is there any other recently proposed method for trajectory optimization? \n\n- For data-augmented MBD, there are also several works that utilize model-free diffusion models to augment data given sub-optimal offline datasets. They claim that through stitching ability, model-free diffusion can also generate novel trajectories [1, 2]. Could authors validate empirically that MBD is more powerful compared to model-free diffusion models given partial and dynamically infeasible datasets?\n\n[1] Ajay, Anurag, et al. \"Is Conditional Generative Modeling all you need for Decision Making?.\" The Eleventh International Conference on Learning Representations. \n\n[2] Li, Guanghe, et al. \"DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching.\" Forty-first International Conference on Machine Learning.", "limitations": "- Please see the weaknesses and questions above.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of the paper is valid. In the trajectory optimization problem, we already know the underlying dynamics of the environment. Therefore, it is natural to utilize such information when we use diffusion models to solve trajectory optimization problems.\n\n- The problem statement is clearly written and easy to follow, even for readers who are not familiar with the topic.\n\n- Experiment results demonstrate that MBD outperforms prior methods in various domains.", "weaknesses": "- In the introduction, the authors claimed that existing diffusion-based approaches often require high-quality demonstration data. However, I cannot find any evidence of experiment results supporting this claim in the manuscript. Diffusion-based planners are not in the baselines of the experiments, and there are several model-free diffusion-based methods that can efficiently generate dynamically feasible and high-rewarding trajectories in RL fields. Could authors validate the reason why model-free diffusion planners are not included in the baselines?\n\n- The authors also claimed that one of the advantages of model-based diffusion over model-free diffusion is its generalization ability (e.g., generalization to new tasks). However, I cannot find empirical evidence for this claim. Could the authors elaborate more on this claim?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the comments and suggestions. We address the comments and suggestions as follows:\n\n---\n\n> In the introduction, the authors claimed that existing diffusion-based approaches often require high-quality demonstration data. However, I cannot find any evidence of experiment results supporting this claim in the manuscript.\n\nSorry for the confusion. As we shown in Sec 5.1, all the tasks is solved by MBD from scratch without any demonstration data, which is the key distinction between MBD and MFD. \n\n---\n\n> Could authors validate the reason why model-free diffusion planners are not included in the baselines?\n\nWe are not comparing with MFD because **MFD is not a direct TO solver but a model-free imitation learning method**. \nFor input, MFD needs large amount of high-quality demonstration data to learn a trajectory.\nIn contrast, MBD is a model-based TO solver that can address TO problems from scratch without relying on demonstration data. Even in cases where demonstrations are used, as discussed in Section 4.3, these demonstrations are partial, feasible, and limited in quantity. Essentially, it is impractical to train a policy effectively with such sparse and low-quality data.\nIn short, the problem setting difference makes comparison between MFD and MBD not meaningful.\n\nPlease refer to General Response 3 for more details.\n\n---\n\n> The authors also claimed that one of the advantages of model-based diffusion over model-free diffusion is its generalization ability (e.g., generalization to new tasks). However, I cannot find empirical evidence for this claim. Could the authors elaborate more on this claim?\n\nThe generalization ability of MBD comes from the fact that MBD *solve* the trajectory given any model and cost function, while MFD *imitate* the trajectory given the demonstration data. \nMBD takes the model and cost function as input, and generate the trajectory satisfying the model and optimal cost function.\nIn contrast, MFD takes the demonstration data as input, and generate the trajectory from the demonstration distribution. \n\nIn other words, for the same modification to the task or dynamics, MBD can re-solve the trajectory with the same algorithm, while MFD needs to re-train the network with the newly collected data under the new setting. \n\n---\n\n> For model-free diffusion, we need to train the score function estimator through the data. In model-based diffusion, which part should be parametrized by the neural network and trained with the data?\n\nPlease refer to general response 3 for the training process clarification. Generally speaking, MBD could be training-free but can be further augmented with data and training. \n\n---\n\n> MBD is compared with the baselines, including zeroth-order optimization and RL algorithms. Is there any other recently proposed method for trajectory optimization?\n\nSolving TO given discontinous hybrid dynamics is generally considered hard, especially for traditional nonlinear programming based solvers like interior point methods, barrier methods, and gradient-based methods which requires the gradient/Hessian information of the dyanmics and costs.\nTo the best of our knowledge, **MBD is the first TO method that can solve full TO problems for high-dimensional tasks with hybrid dynamics** thanks to the diffusion-style optimization process.\nThose tasks are used to be considered to be solved by RL algorithms with a large amount of data.\nThat's why we not only compare with other TO methods, but also add RL as a performance reference.\n\n---\n\n> For data-augmented MBD, there are also several works that utilize model-free diffusion models to augment data given sub-optimal offline datasets. They claim that through stitching ability, model-free diffusion can also generate novel trajectories [1, 2]. Could authors validate empirically that MBD is more powerful compared to model-free diffusion models given partial and dynamically infeasible datasets?\n\n**Comparing MBD with MFD is not apples-to-apples comparison due to their setting difference**. \nThe major difference lies in the problem setting: MBD is a TO solver while [1,2] is offline RL algorithms.\nIn other words, in terms of data availability, MBD does not require any data (but can be augmented with data), while [1,2] require large amount of offline dataset like D4RL.\nBesides, the dataset like D4RL used in [1,2] is always dynamically feasible (even though it might not be optimal), while MBD can easily handle dynamically infeasible trajectory.\nIn our experiments, the demo data is not dynamically feasible so [1,2] can not even be applied, whereas MBD successfully integrates the demo data.\nIn short, it is not meaningful to compare MBD with [1,2] due to the difference in input and the data requirement of MBD is much weaker than [1,2].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We thank all reviewers for their valuable comments.
We want to make the following clarifications and improvements to the manuscript:

## 1. Clarify the Problem Setting when Comparing with RL

We include **PPO/SAC as a performance reference not as a baseline** because there is no existing TO method that can solve such high-dimensional discontinuous tasks as we have shown in the experiments.
Model-free RL, especially PPO/SAC, is widely used in such tasks and is considered the SOTA method. 

As pointed out by reviewer `sfi931`, RL differs from MBD in terms of **optimization objectives** (the horizon difference) and **optimization variables** (optimize for a policy v.s. a trajectory). We address the concern as follows:

**optimization objective**: To clarify the effect of different objectives, especially the horizon difference, we conducted an ablation study by swapping the optimization objectives of MBD and RL. RL optimizes for a longer horizon discounted reward $J = \sum_{t=0}^{H_\text{RL}} \gamma^t r_t, H=1000, \gamma<1$ while MBD optimizes for a shorter horizon undiscounted cumulative rewards $J = \sum_{t=0}^{H_\text{MBD}} r_t, H=50$, $\gamma=1$. Here we compare the performance of MBD and RL under each other's optimization objectives:

Table1: Mean Reward Across All Tasks of MBD and RL under TO and RL objectives
| Algo.| RL  |  MBD |
| - | - | - |
| RL Objective ($H=1000, \gamma<1$) |  1.33   | 1.93  |
| TO Objective ($H=50, \gamma=1$) |   0.23  | 2.12  |

As shown in Table 1 and Figure 1 in the accompanying PDF, 
MBD outperforms RL by 44.5% under the RL objective and 805.5% under the TO objective. 
The results demonstrate that MBD's superior performance is attributed to its better diffusion-style iterative optimization process compared with RL's random exploration.

**optimization variables**: As mentioned by reviewer `sfi913` and `X3LL`, RL optimizes for an online policy while MBD optimizes for an offline trajectory. Actually, we can easily extend offline MBD to online settings with a receding horizon strategy. We will address that in the next section. 

## 2. Extension to Closed-loop Control with Receding Horizon

MBD is designed to be a TO solver, whose output is a trajectory rather than a policy.
That's why we only evaluate the open-loop MBD which represents the trajectory optimization capability of MBD.

We fully acknowledge reviewer `sfi913` and `X3LL`'s concerns in extending MBD to online control.
In short, **there is no fundamental limitation to extending MBD to closed-loop control with a receding horizon thanks to our fully model-based nature**. That means MBD can be easily conditioned on each step's observation and recalculate the trajectory in a receding horizon manner. Here is the naive receding horizon MBD algorithm:

```
Alg: MBD with Receding Horizon
Init: Optimize trajectory $x_{0:T}, u_{0:T-1}$ with MBD
For t=0 to T-1:
    Observe the state x_t
    Optimize trajectory $x_{t:T+t+1}, u_{t:T+t}$ with single-step MBD
    Apply the first control input $u_t$ to the system
    Shift the trajectory forward to provide an initialization for next step $x_{t+1:T+t+2}, u_{t+1:T+t+1}$
End For
```

Table 2: Mean Reward Across All Tasks of MBD (open-loop), MBD (receding) and RL
| RL | MBD(receding) | MBD |
| - | - | - |
| 1.33 | 2.32 | 2.12 |

Thanks to the closed-loop control and online optimization, 
the results in Table 2 and Figure 2 in the accompanying PDF show that **the most naive receding horizon MBD can further improve MBD's performance by 9.6%** compared to the open-loop MBD. 

## 3. The Training Process Clarification of MBD

**TL;DR: MBD does not involve explicit training components but can be further improved with extra data**.

Although MBD and MFD share the same iterative refinement process, MBD's setting is fundamentally different from generative model while more similar to MPC, where the dynamic model and constraints are well-defined but solving the problem is challenging. This difference also leads to the online sampling-based score computation from the model without off-line approximating it with a neural network.

This clarification should address concerns raised by reviewers `YGTC` and `sfi913` regarding the model’s capability for conditional planning. As our method is entirely model-based, it can adapt to any feasible observation and optimize trajectories without additional data collection or model retraining.

Acknowledging reviewer `X3LL`'s points, we agree on the value of introducing a learned module into the framework.  A potential future direction could involve integrating learning dynamics and cost functions, akin to model-based RL. We intend to explore this as a future direction and will note it in the revised manuscript.


## 4. Robustness and Generalization of MBD

Even though MBD only generates open-loop trajectories, **MBD is also robust to modeling error thanks to its iterative refinement process following the backward process of diffusion**.
In answering reviewer `X3LL`'s question about the robustness of MBD, we compare the performance of MBD with RL algorithms under 0.05 stochastic control noise in Figure 3 in the accompanying PDF.

Table 3: Mean Reward Across All Tasks of MBD and RL under Noisy Setting
| Algo. | RL| MBD(receding) | MBD |
| - | - | - | - |
| w/o noise | 1.33 | 2.32 | 2.12 |
| w/ noise | 1.00 | 2.23 | 1.65 |

As shown in Table 3 and Figure 2 in the accompanying PDF, adding noise does not lead to catastrophic failure in MBD, while the receding horizon version of MBD still outperforms RL by $65.3\%$ across all tasks under the noisy setting.
By refining the trajectory iteratively, MBD can effectively handle the noise and modeling error, making it more robust than RL.
Actually, in practice, standard trajectory output can be further fine-tuned with the updated model with data, which could further improve the performance of MBD given imperfect models.

Meta-review: ### Summary
The paper introduces a method called Model-Based Diffusion (MBD) for solving trajectory optimization (TO) problems. Unlike traditional model-free diffusion approaches, MBD leverages explicit model information to compute the score function in the diffusion process. This allows it to efficiently solve TO problems without relying on pre-existing data. The method also has the flexibility to incorporate data of varying quality, enhancing its applicability in diverse tasks. Experimental results demonstrate that MBD surpasses state-of-the-art methods, including reinforcement learning and sampling-based optimization, particularly in complex tasks involving contact-rich environments. The paper also highlights MBD's ability to integrate incomplete or imperfect data, offering practical advantages over standard diffusion models.

### Decision
The paper is well-written, and the application of model-based diffusion for the trajectory optimization problems is novel and interesting. The reviewers initially had concerns about the paper on the applicability of the proposed method to real-world problems, experiments (in particular, comparison to simple RL baseline like PPO), and clarity of the theory section. However, I think during the rebuttal period, the authors have done a good job addressing most of these concerns. The paper was borderline. However, the approach is novel, and there are no critical problems, the concerns raised by the reviewers could be addressed with a minor modification of the paper. I recommend the authors to carefully read the points made by the reviewers and clarify those points for the camera-ready version of the paper.
Decision: Accept (poster)