Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching
Authors: H.J. Terry Suh
Glen Chou
Hongkai Dai
Lujie Yang
Abhishek Gupta
Russ Tedrake
Venue: CoRL 2023 Poster (2023)
Keywords: Model-Based Reinforcement Learning
Offline Reinforcement Learning
Planning under Uncertainty
Diffusion
Score Matching

Abstract: Gradient-based methods enable efficient search capabilities in high dimensions. However, in order to apply them effectively in offline optimization paradigms such as offline Reinforcement Learning (RL) or Imitation Learning (IL), we require a more careful consideration of how uncertainty estimation interplays with first-order methods that attempt to minimize them. We study smoothed distance to data as an uncertainty metric, and claim that it has two beneficial properties: (i) it allows gradient-based methods that attempt to minimize uncertainty to drive iterates to data as smoothing is annealed, and (ii) it facilitates analysis of model bias with Lipschitz constants. As distance to data can be expensive to compute online, we consider settings where we need amortize this computation. Instead of learning the distance however, we propose to learn its gradients directly as an oracle for first-order optimizers. We show these gradients can be efficiently learned with score-matching techniques by leveraging the equivalence between distance to data and data likelihood. Using this insight, we propose Score-Guided Planning (SGP), a planning algorithm for offline RL that utilizes score-matching to enable first-order planning in high-dimensional problems, where zeroth-order methods were unable to scale, and ensembles were unable to overcome local minima. Website: https://sites.google.com/view/score-guided-planning/home

Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching
H.J. Terry Suh CSAIL,, MIT Cambridge, MA 02139 hjsuh@mit... edu
Glen Chou" CS MIT Cambridge, MA 02139 gchoušmit, edu
Hongkai Dai Toyota Research Institute Los Altos, CA 94022 hongkai wwwidaiiriiii wew.etirrllbbal.. global
Lujie Yang" CSAIL, MIT Cambridge, MA 02139 lufie@mit.edd teteemiiteeee edu
Abhishek Gupta University of Washington Seattle, WA 98195
Russ Tedrake CSAIL, MIT Cambridge, MA 02139 russtêmit t. edu
abhgupta@cs.washie edu
Abstract: Gradient-based methods enable efficient search capabilities high dimensions. However, in order to apply them effectively in offline optimization paradigms such offline Reinforcement Learning (RL) or Imitation Learning (IL), we require more careful consideration of how uncertainty estimation interplays with first-order methods that attempt to minimize them. We study smoothed distance to data as an uncertainty metric, and claim that has two beneficial properties: (i) it allows gradient-basee beaseeeett-eedddddee methods that attempt to minimize uncertainty drive iterates to data as smoothing annealed, and (ii) it facilitates analysis of model bias with Lipschitz constants. As distance data be expensive compute online, we consider settings where need amortize this computation. Instead of learning the distance however, we propose to learn its gradients directly an oracle for first-order optimizers. We show these gradients can be efficiently learned with score-matching techniques by leveraging the equivalence between distance to data and data likelihood. Using this insight, we propose Score-Guided Planning (SGP), a planning algorithm for offline RL that utilizes enable first-order planning in high-dimensional problems, where zeroth-order methods unable to scale, and ensembles unable to overcome local minima. Website: http:///we..oooee..oo///////e//e
matching
Keywords: Diffusion, Score-Matching, Offline, Model-Based Reinforcement Learning, Imitation Learning, Planning under Uncertainty
1 Introduction
Uncertainty minimization central problem offline optimization, which manifests as many different paradigms in robot learning. offline model-based RL (MBRL [1, 2, 3. 4)), penalization of uncertainty acts regularizer against model bias [5, 6, 7] and prevents the optimizer from exploiting model error [8, 6, 9, 10]. In offline model-free RL, it regularizes against overestimation of the Q function [11, 12]. In addition, many imitation learning (IL) algorithms can be viewed as minimizing distribution shift from the demonstrator distribution [13].
Despite the importance of uncertainty, statistical uncertainty quantification remains a difficult problem [14, 15, 8, 6, 16]; Gaussian processes (GPs) [6, 17, 18] rarely scale to high dimensions, and ensembles [8, 4, 19] taken the approach of staying near the data by maximizing data likelihood. These methods either minimize distribution shift between the optimized and data distribution for behavior regularization
prone to underestimating true uncertainty [15]. As such, previous works have often
* These authors contributed equally to this work.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

[20, 21, 22, 23], or the occupation measure of the optimized policy and the data distribution [11, 24, 25, 26, 27, 28, 29, 30]. Both of these require the likelihood of the data to be estimated.
promising direction for estimating the data likelihood to leverage techniques from likelihood- based generative modeling, such as variational autoencoders (VAE) [31, 32. 33], generative adversarial networks (GAN) [34, 35, 36, 13], and flow-baser models [25, 37J. Yet, these prior works have shown that training density models to generate accurate likelihoods can be challenging, especially for high- dimensional data. While gradient-basse -ased methods have good scalability properties that make them desirable for tackling offline optimization problems in this high-dimensional regime, the effect of incorrect likelihoods are further exacerbated in this setting as they lead gradient-based methods into spurious local minima. Thus, we ask in this work: can we design gradient- based offline optimization methods that encourage data likelihood, without explicit generative modeling of likelihoods?
Our key insight is that in order to maximize the data likelihood with gradient-based based methods, we do not need access to the likelihood itself. Rather, having access to first-order oracle (gradients), known as the score function, is sufficient. We claim that directly utilizing the score function has two benefits compared to likelihood- based modeling. i) First, recent breakthroughs in score-based modeling [38, 39, 40] show that the score function is considerably easier to estimate with score-matching techniques [40, 39], it bypasses estimation of the partition function that is required for computation of exact likelihoods [40, 41]. ii) In addition, show that score matching with annealed perturbations [40] gives gradients that stably drive decision variables to land exactly on data when uncertainty is minimized with gradient- based optimization, property we term data stability. We demonstrate this by showing that the negative log likelihood of the perturbed empirical distribution, whose gradients score-matching estimates, is equivalent to a softened distance to data [42].
Furthermore, ask: when, and why, would approaches that penalize distance data surpass the ensemble method of statistical uncertainty quantification [4, 9]? We show that unlike empirical variance among ensembles, we can relate how much smoothed distance to data underestimates true uncertainty with the Lipschitz constant, for which we can use statistical estimation to put confidence bounds [18], or utilize structured domain knowledge [43]. Morcover, we show that ensembles do not necessarily have the data stability property due to statistical noise; therefore, optimizing for ensemble variance can easily lead to local minima away from data in gradient- based optimization.
To put our theory into practical algorithm, we propose Score-Guided Planning (SGP), gradient- based algorithm that estimates gradients of the log likelihood with score matching, and solves uncertainty-penalizee offfine optimization problems that additively combine the cumulative reward and the log likelihood of data without any explicit modeling of likelihood. SGP enables stable uncertainty minimization in high-dimensional problems, enabling offline MBRL to scale even to pixel action-spaces. We validate our theory on empirical examples such as the cart-pole system, the D4RL benchmark [44], a pixcl-space single integrator, and box-pushing task [45] on hardware.
2 Preliminaries
Offline Model-Based Optimization. We first introduce setting of offline model-based optimiza- tion [46]. In this setting, we aim to find x that minimizes an objective function f R. + R, but are not directly given access to f; instead, we have access to 2; ~ p(x). and their corresponding values f(x)). such that the dataset consists of D = {(z,, f(z))) Denoting p(x; D) as the empirical distribution corresponding dataset D. offline model-based optimization solves
min for (x) s.t. g" = arg min EerD(::)
8
In words, minimize surrogate loss fx(x,, where choose 0 as the solution empirical risk minimization of matching f given the data. Denoting x arg ming fo- (z), one measure of performance of this procedure is error at optimality, f[ - fo(x")ll
Uncertainty Penalization. The gap llf(==) fo(x")ll potentially be large if fe fails to approximate f correctly at x", which is likely if xt is out-of-distribttion (o.o.d)). To remedy this,

---

previous works [8, 46, 47] have proposed adding a loss term that penalizes o.o.d. regions. We denote this penalized objective as
fo(æ) = fo(x) + (x))
(2)
where 3 E 1R20 is some weighting parameter, and μ((x) is some notion of uncertainty, Intuitively this restricts the choice of optimal z to the training distribution used to find 8", since these are the values that can be trusted. If the uncertainty metric overestimates the true uncertainty, ll. - fo(x)) z
ux(:), it possible bound the error optimality directly using ju(=").
Offline Model-Based RL. While the offline model-based optimization problem described is one-step problem, the problem of offline model-based RL (MBRL) involves sequential decision making using an offline dataset. In offline MBRI,, we are given dataset 'D = { Me (t+11)+)) where x E IR" denotes the state, u E I denotes action, is the time index and is the sample index. Again denoting p(,, ue #t+1; D) as the empirical distribution corresponding to D, and introducing pl,, uz) as a state-action uncertainty metric [8], eneertainty-pnnaliinn offline MBRL solves
Σ re( uz) - By² (xre, uz)
max #1:7
T++1 fo- (It, Us) Vt. 0" arg min me [|lae=1 So (2x, u.) uz]
In words, we first approximate the transition dynamics from data, then solve the uncertainty pennaliie optimal control problem assuming the learned dynamics. Previous works have used ensembles [8, 9, 48], or fikelihood-basee generative models of data [25] to estimate this uncertainty.
The resulting open-loop planning problem can either be solved with first-order methods [49, 50], or zeroth-order sampling-bass -ased methods [45, 9] such as CEM [51] or MPPI [52]. While insights from stochastic optimization [53] tell us that first-order methods are more favorable in high dimensions and longer horizons as the variance of zeroth-order methods scale with dimension of decision variables [54], first-order methods require careful consideration of how amenable the uncertainty metric is to gradient-based optimization.
3 Distance to Data as a Metric of Uncertainty
In order to find a metric of uncertainty that is amenable for gradient-based offline optimization, we investigate smoothed distance to data as a candidate. We show that this metric allows us to drive iterates gradient- based methods that minimize uncertainty to land on data as smoothing annealed. In addition, it allows us quantify how much we underestimate true uncertainty by using the Lipschitz constant of error. All proofs for theorems in this section included in Appendix A.
3.1 Properties of Distance to Data
We first formally define our proposed metric of uncertainty for offline model-based optimization.
Definition 3.1 (Distance to Data). Consider dataset D = {xx) } and an arbitrary point x E RR. The standard squared distance from æ to the set D can be written as d(z; D)2 = ming:eD 1|l - x 1|2 We define smoothed distance to data using a smoothed version of this standard squared distance,
(4) where C' is some constant to ensure positiveness of (x; 2) and > 0, also known as the temperature parameter, controls the degree of smoothing with a - 0 converging to the true min.
d D)2 Softmin,, } z2,1² C' -2 log exp [- ب || 2111 +C,
The motivation for introducing smoothing is to make the original non-smooth distance metric more amenable for gradient-based optimization [55, 56]. We now consider benefits of using u(x) do (x; D) as our uncertainty metric, and show that as the smoothing level annealed down, minimizing this distance allows to converge the points in the dataset.
Proposition 1 (Data Stability). Consider monotonically decreasing sequence σμ such that d -+ 0. and denote 26 the n gradient descent iteration of min, dog (2; 2)2. Then, almost surely with
3

---

random initialization and appropriate step size, we have
lim lim ox E D. R-200 n0000
(5)
We note that empirical variance among ensembles [8] prone having local minima away from data due to statistical variations, especially with small number of ensembles, which makes them unreliable for gradient- baased optimization. Next, we show that it is possible to analyze how much softened distance to data underestimates true uncertainty using the Lipschitz constant of the model bias Le- Proposition (Lipschitz Bounds), Let Le be the local Lipschitz [57] constant of the true error (bias) e(x) = (x) fo (z)ll2 valid over Z S X, where X' the domain of the input z. Then, e(z) is bounded by
e(x) S e(x.)) v2L. /d. (x; D)2 C2,
(6)
for all x E 2, where Ze = arg min 1 a|| x:ll2. i... the closest data-point, and C2 o² log - C, where C' is defined (4).
In general, it difficult obtain Lg in the absence of more structured knowledge of f. However, possible to obtain confidence bounds on L. using statistical estimation with pairwise finite slopes ((lx))ll// )1 ||; æy within the dataset [58, Ch. [18]. We believe this offers benefits over ensembles as characterizing the convergence of neural network weights with randomly initialized points is far more complex to analyze. We compare distance to data to other uncertainty metrics in Figure in simple ID offline model-based optimization, where we show that ensembles [8, 9] have local minima outside of data, and unpredictably underestimates uncertainty due to model bias.
[[e(x.) - e(x;))
3.2 Estimating Gradients of Distance to Data with Score Matching
Although we have shown benefits of smoothed distance to data as an uncertainty metric amenable for gradient-based optimization, is costly to compute at inference time as we need to iterate through the entire dataset. At the cost of losing guarantees of exact computation in Section 3.1, we consider ways amortize this computation with function approximation. We first show the equivalence of the smoothed distance-to-das -t--ttaatt- to the negative log likelihood of the perturbed empirical distribution [40], which applies randomized smoothing [55, 56] the empirical distribution p(x; D).
Definition 3.2 (Perturbed Empirical Distribution). Consider a dataset D = {a:} and its corresponding empirical distribution pa;; D). We define po (z; D) as the noise-perturbed empirical distribution, Po (x D) p(æ; (x'; o² I² Ex.eD N(o;; x,, o²,,
Processes (GP)
Ensembles
2-
Predicted
Bound (xi) a ppxx)
0-
Squared Distance to Data
GP Variance
Ensemble Variance
B = 0.5
15.0
8.0
15.0
Uncertainty Penalty uód?
12.5
12.5
Error lddxt - fxxp
80
10.0
10.0
β 400.0
5.0
5.0
0-
0.0
0.0
Figure 1: Comparison of different uncertainty metrics. Top row: Visualization of distance to data against GPs and ensembles with M = 2. Bottom row: Visualization of the penalty μxz)²² All the metrics underestimate uncertainty to varying degrees, but distance to data can be analysis; as we increase samples, distance to data will be able to bound the true uncertainty more closely by estimating Lipschitz constants using pairwise slopes in the dataset. In addition, distance data shows more stability to data while ensembles have local minima outside of data.
amenable for

---

Proposition 3. The negative log- logllikelihood of the perturbed empirical distribution Pa (z) is equivalent to smoothed distance to data by factor of o², up to some constant that does not depend on x,
-o² log Po (z; D) do (z; D)² + n(,,n,o).
(8)
This connection the perturbed empirical distribution allows us generative modeling tools that randomly perturb data [59, 40, 39], such as denoising autoencoders. However, computing likelihoods directly have shown to be difficult for high-dimensional data [40]. Additionally, even if such a likelihood-based generative model can estimate likelihoods with low error, there are no guarantees that its gradients will also be of low error (Appendix A.4), which defeats our purpose of finding a metric amenable to gradient- baasentt-baedddd optimization. Thus, we propose to estimate the gradients of the perturbed empirical distribution (score function) directly [40, 39], which have shown promising performance in generative modeling [38] bypasses the estimation of the partition function.
Estimating the score function is a process known as score-matching. Following [40], we introduce coise-conditioned score function s sz σ) = Va log Po (z; D) [40] and aim to optimize the following objective given some decreasing sequence of annealed smoothing parameters dg.
min Σn of Exxpe (a:D) [[]s0 (2, oR)) v log (2;; D)
which has been shown to equivalent to the denoising seoes-cooeeeeeeeee motoe-nttchig loss [59]. Compared to explicitly computing v log Poz (z; D) which would require iterating through the entire dataset, the denoising loss allows us to learn the score function using batches of data.
min Σx of
anD(a;D) [1lso(a(;<x)) + 3 =)] a'~N(a' (1)
(10)
4 Planning with Gradients of Data Likelihood
Having illustrated benefits of using smoothed distance to data as an uncertainty metric, we now turn to the sequential decision-making makinn setting of offline MBRL, where we use notation from Section 2.
Offline MBRL with Data Likelihood. Consider a learned dynamics model fa from the dataset D {Z U4 Fe+1 ) as well the perturbed empirical distribution Por (Z,, uy; D) of the (z,, u,) pairs the dataset D. Then, given some sequence rewards Tt, we consider the following planning problem of maximizing both the reward and the likelihood of data,
Eralan ue) + B0
Σ log (21, D)
max 21:7 M1:T
(11)
s.t. 1r41 So(x,, u.) AI E [1,7].
Note that -o² log Po (2e, uz) is equivalent to smoothed distance to data d, (z,, ue: D)2 as an objective, and acts as an uncertainty penalty, preventing the optimizer from exploiting o.o.d. solutions.
Score-Guided Planning (SGP). Given differentiable rt, we propose single-shooting algorithm that rolls out fe and computes gradients of Equation (11) with respect to the input trajectory. We first denote the sensitivity of the state and input at time (æz, un) with respect to the input at time j as 0x,/Ouj and 0u,0uu, Then, the gradicnts of Equation (1I) can be expressed using this sensitivity,
T
Σ
re ue) + Bo² log Po (xe, ue;
Ouj
(12)
Σ
T
Dr. Ou Oer, Üuy Du, Ou, + 30²
Ör. Ox
① 1og pa (21, we)
Ox.
ô log pa (2 uz)
Ou:
Or
Oug
Ou.
Dug
Note that Ô log Po (Ie, 00x and 0 log (Iz, uz)/Ou, can be obtained using the noise-conditioned score estimator s0(xr, u, o) in Section 3.2, where we extend the domain to include both x and After using reverse-mode autodiff (Appendix B. compute the gradient, we call optimizers that accept gradient oracles, such as Adam [50]. We also note that this gradient computation can be extended
5

---

to feedback policy gradients in Appendix B.5. Finally, we train a poiss-connitioned score function for some sequence oe and anneal the noise-level during optimization following [40]. SGP is also described with an algorithm block in Appendix B.3.
Related Methods. LDM, COMBO, and DOGE [25, 12, 60] closely-related offline methods that penalize for data likelihood distance to data. Many IL methods [41] are also related, as SGP with zero rewards can be used to imitate demonstration data by maximizing data likelihood [13] (Appendix B.6). In particular, AIRL [35] maximizes a rewardless version of our objective using GANs[34]. However, these methods primarily rely on likelihood- based generative models, and do not consider the interplay of generative modeling with gradient- bbsed optimization. Diffuser [26, 61] shares similar methodologies with SGP, and solves a variant of Equation (11) with a quadratic- penalty-based approximation of direct transcription (Appendix B.4), which comes with benefits of numerical stability for long horizons [62, exercise 10.1], and robustness to sparse rewards [63].
5 Empirical Results
We now test our proposed algorithm (SGP) empirically and show that it is an effective method for offline optimization that has scalable properties in high-dimensional problems by leveraging gradient-based optimization. All details for the included environments are presented in Appendix C.
Cart-pole System with Learned Dynamics. We apply our method to swing up a cart-pole model, which undergoes a long duration (T = 60) dynamic motion, which directly translates to high number of decision variables in single shooting. We first show the effects of model bias in Fig. 2.C, where vanilla MBRL plans trajectory that leaves the region (red box) of training data. In contrast, SGP keeps the trajectory within the training distribution 2.A). We further compare method against the baselines of ensembles (Fig.2.B,E) and CEM [51] (Fig.2.D.E) To fairly compare CEM against SGP, we additionally train data distance estimator that predicts the likelihoods directly based on explicit computation of the softmin distance in Equation (4) (Appendix C).
We observe that the convergence of CEM is much slower than first-order methods (Fig.2.F). However, surprisingly obtain more asymptotic performance with CEM rather than using Adam for ensembles we believe this signifies presence of local minima in pradient-based minimization of ensemble variance, opposed to CEM which has some stochastic smoothing that allows it to escape local minima [54]. Finally, we note that unlike score functions, distance to data is considerably more difficult to train as we need to loop through the entire dataset to compute one sample. As a result, it is costly to train and we were unable to train it to good performance in a reasonable amount of time.
Function (SGP)
Adam Ensemble
(Vanila MBRL)
plant
plan
10
pinnt
32
A
-3
3
B
8
-50
-20
-10
CEM DataDstance
CEM Ersembbe
lterations
B-0.0
101
plan
Sin
Adam with Scorefurction
10
Adam with onsemble. CIN with ensembble CEN with DetaDistance
pla
D
-30 10
Tee seftnn
20000
56000
Figure 2: Experiments for the cart-pole system, where blue is the hallucinated plan and green is the actual rollout, and the red box is the regime of collected data. Note that SGP (A) results in successful swingup while MBRL (C) fails. Compared to other baselines such as B.D.E, our method achieves much lower true cost with faster convergence (F).

---

Dataset
Env
BC
MOPO
MBOP
AWAC
CQL
Diffuser
Ours
Random
halfcheetah
2.1
35.4
6.31 400
35.4
31.3
Random
hopper
1.6
11.7=00
10.8+0.3
9.6
10.8
12.3=1.3
Random
walker2d
9.8
13,6#26
8.1+5.5
5.1
7.0
15.6=7.1
Medium
halfcheetah
42.6
1..311.6
44.61108
37.4
44.0
42.8.1.3
46.6
Medium
hopper
52.9
28.0
48.81126.
72.0
58.5
1..
60.6
Medium
walker2d
75.3
17.8+19.3
41,0+:29.4
30.1
72.5
79.6:10.6
36.2=6688
Med-Exp
halfcheetah
55.2
63.3
105,9:1778
36.8
91.6
88.940.
61.6=24
Med-Exp
hopper
52.5
23.74660
55.1+:44.
80.9
105.4
103.341.
109.2-0.3
Med-Exp
walker2d
107.5
44.6:1229
70.24:36.2
42.7
108.8
106.9:022
103.0=28
Table I: Comparing performance of our algorithm on the D4RLL [44] dataset. Similar to [26, 68), we
bold-face 95% of max performance. Ours averaged from 5 trials. We emphasize that we outperform ensembles (MOPO) in many of the tasks.
D4RL Benchmark. To evaluate our method against other methods on standard benchmark, we use the MuloCo [64] tasks in the D4RL [44] dataset with three different environments and sources of data. To turn our planner into a controller, we solve the planning problem with some finite horizon, rollout the first optimal action "i, and recompute the plan standard model-predictive control (MPC) [65] fashion. We compare against methods such as Behavior Cloning (BC) [22, 21], MOPO [8], MBOP [66], CQL [23], AWAC [67], and Diffuser [26]. Our results demonstrate that algorithm performs comparably to other state-of-the-ari methods. On many of the tasks, demonstrate better performance compared to MOPO [8] which uses ensembles for uncertainty estimation, while requiring less memory. This empirically supports our proposed benefits of using score matching for gradient-based offline MBRL. In addition, we outperform BC in many tasks, illustrating that we achieve better performance than pure imitation learning by incorporating rewards.
Pixel-Based Single Integrator. Pixel spaces have long been a challenge for MBRL due to the challenges of reliably controlling from images [69], learning pixel-space dynamics, and combating the resulting model-bias [2. 30, 70, 71]. To show scalability of our method, we present 32 32 pixel-based single integrator environment where the action space is defined in pixel space, similar to spatial action map [72]; Fig. 3 for visuals. Our planning problem requires proposing sequence of 32 x 32 control images, which minimizes cost (goal- reaching while minimizing running cost) when rolled out through the dynamics (see Appendix C for details and numerical results). We demonstrate that SGP capably mitigates model bias, generating plans consisting of plausible observation/controls near the data; thus, these plans are closely followed when executed open-loop under the true dynamics (Fig.3.A). When ignoring the effect of model bias and planning naïvely with MBRL (Fig.3.B), i.e.. setting A = 0, the resulting trajectory exploits the subtleties of the chosen cost function, planning unrealistic trajectory which leads to high cost at runtime. Also, we show that ensembles are unable to stably converge back to the data manifold of images seen during training, instead getting trapped in local minima, planning unrealistic trajectories which translate to poor runtime rollouts on the true system (Fig.3.C). Moreover, due to the high dimensionality of this action space (u, E IR322, ), we see that zeroth-order methods, such as CEM [51], lead to very low convergence rates (Fig.3.D) due to the large number of decision variables [53, 54]; again, this leads to high cost seen at runtime.
Planned observations Imm
Planned controls
True observations T
Acaan 30P
T=
1-14
7
T
T
Srow
T-3
NoM MMRL
T
T-3
T
T-9
T 12
14
T
T-3
T
T
T
Man Eruendi 19 18
CE4
IM
Figure 3: Planned (left) and true (right) rollouts of observations for the pixel single-integrator with pixel action space (center). Note that SGP (A) significantly outperforms Vanilla MBRL (B), Adam +
Ensembles (C), and CEM Ensembles (D), which are unable to combat model bias during rollout.
7

---

Hardware Selup
Uncertainty Minimization
Predicted Trajectory
MBRL Predicted Trajectory
B
A
enater Dessera Tiaaaaar
C
D
Demonstration atlon Data
Goal: Down
Goal: Middle
Goal: Up
Tijapaty
Marter tujdbryttan Audin supobo prans)
Moder Fraacey
Matir hugikky pa3r
F
G
Husel: (4041 Made Medb--Peodoors
H
Figure 4: Visualization of the box-pushing experiment in A. where data was collected from 100 demonstrations illustrated in E. We show that SGP successfully stabilizes markers back to their implicit constraints (B), which allows better predictions (C) than vanilla MBRL (D). We additionally use the reward to change the goal into three different regions of down (F), middle (G), and up (H), where vanilla MBRL fails, and behavior cloning not capable of.
Box-Pushing with Learned Marker Dynamics. To validate our method on the data-scarce regime of hardware, we prepare a box-pushing task from [45] where we leverage the quasistatic assumption [73] and treat the positions of the motion capture markers as if they are states. An interesting feature of this setup is that the markers implicitly live on a constraint manifold where the distance between each marker is fixed; we ask if using score matching can stabilize the rollouts of prediction to obey this implicit constraint, similar to how diffusion stabilizes back to the data manifold [42]. To test this, we collect about 100 demonstrations of the box being pushed to different positions, which lead to 750 samples of data tuples (I,, Me, T++ 1). We aim to show that i) SGP can enforce implicit constraints within the data, ii) distance to data acts as successful uncertainty metric in data-scarce offline MBRL, and iii) we can use the reward to change the task from imitation data.
Our results in Fig.4 4 demonstrate that SGP successfully imposes implicit constraints on the data. Not only does minimization of uncertainty in the absence of rewards result in stabilization the marker position constraints (Fig.4.B), but the rollouts also become considerably more stable when we use the distance to data penalty (Fig.4.C). In contrast, MBRL with 3 = O destroys the keypoint structure as dynamics are rolled out, resulting in suboptimal performance (Fig.4.D). Finally, we demonstrate through Fig.4.F,GG G.H that we can use rewards to show goal-driven behavior to various goals from a single set of demonstration data, which behavior cloning is not capable of.
6 Conclusion and Discussion of Limitations
We proposed SGP, which is a first-order offline MBRL planning algorithm that learns gradients of distance to data with cote-nnt techniques, and solves planning problems that jointly maximize reward and data likelihood. Through empirical experiments, of zeroth-order methods and ensembles, has comparable performance with state-of-art offline RL algorithms, and scales to pixel-space action spaces with up to 15, 360 decision variables for planning.
showed that SGP beats baselines
We conclude with listing some limitations of our approach. Unlike ensembles, our method by construction discourages extrapolation, which can be a limitation when the networks jointly recover meaningful inductive bias. We also believe that computation is current bottleneck for realtime appli- cation of our MPC, which can take between 1-2 seconds per iteration due to gradient computations. Finally, we have not investigated the performance of SGP under aleatoric uncertainty [48].
8

---

Acknowledgments
This work was funded by Amazon.com Services I.LC; Award No. PO #2DD003300336 and Office of Naval Research (ONR); Award No. N00014-22-1-1121. We would like to thank Pulkit Agrawal and Max Simchowitz for helpful discussions on the paper.
References
[T] C. Atkeson and Santamaria. A comparison of direct and model-based reinforcement learning. In Proceedings of International Conference Robotics and Automation, volume 4, pages 3557-3564 vol.4, 1997. doi: 199 6666966.....
[2] H. T. Suh and Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. S. M. LaValle, M. Lin, T. Ojala, D. Shell, and Yu, editors, Algorithmic Foundations of Robotics XIV, pages 347-363, Cham, 2021. Springer International Publishing. ISBN 978-3-030-66622-88
[3] M. Watter, J. T. Springenberg. Boedecker, and M. Riedmiller. Embed to control: locally linear latent dynamics model for control from raw images, 2015.
[4] K. Chua, R. Calandra, R. McAllister, and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models, 2018.
[5] Schneider. Exploiting model uncertainty estimates for safe dynamic control learning. In
M. Mozer, M. Jordan, and Petsche, editors, Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996. URL URR..tttps:/pprcceeedngg neurips. cc/paper_ http:///pppppe...e..///////ttt/r
Paper ppp.r.
[6] M. Deisenroth and C. Rasmussen. Pilco: A model-based and data-efficient approach policy search. pages 465-472, 01 2011.
[7] Abbeel, M. Quigley, and A. Y. Ng. Using inaccurate models reinforcement learning. In Proceedings of the 23rd International Conference on Machine Learning, ICML, '06, page 1-8, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: Noi:10.1155/1448444 143845. URL https: //doi. 1ttp//..114/11148800 1143845.
[8] Yu, G. Thomas, L. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization, 2020.
[9] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In L. P. Kaclbling. D. Kragic, and K. Sugiura, editors, 3rd Annual Conference Robot Learning, CoRL. 2019, Osaka, Japan, October 30 November 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 1101-1112. PMLR, 2019, URL
mlr, watpa///vv10//aaaaannanaa htmll html.
[10] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel Model-based offline reinforcement learning, 2021.
[11] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration, 2019.
[12] Yu, A. Kumar, R. Rafailov, A. Rajeswaran, Levine, and C. Finn. Combo: Conservative of- fline model-based policy optimization. M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 28954-28967, Curran Associates, Inc., 2021. URL http:tttt///ppceeeddinge meurtiiiiiiiininnsssseeeeeeeeiee cc/ .df.
[13] K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective imitation learning methods, 2019.

---

[14] A. Malik, V. Kuleshov, J. Song, D. Nemer, H. Seymour, and S. Ermon. Calibrated model-based deep reinforcement learning. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4314-4323, PMLR, 09-15 Jun 2019. URL https://prooceedings mlr. pptpps//9/emmlaleeeeea html.
[15] V. Kuleshov, N. Fenner, and S. Ermon. Accurate uncertainties for deep learning using calibrated regression, 2018.
[16] Y. Gal and Z. Ghahramani. A theoretically grounded application of dropout in recurrent neural networks, 2016.
[17] J. Kocijan, R. Murray-Smith, C. Rasmussen, and A. Girard. Gaussian process model based predictive control. In Proceedings of the 2004 American Control Conference, volume 3, pages 2214-2219 vol.3, 2004. 16::10022919/ACC20004..33
[18] C. Knuth, G. Chou, N. Ozay, and D. Berenson. Planning with learned dynamics: Probabilistic guarantees on safety and reachability via lipschitz constants. IEEE Robotics Autom. Lett., 6(3): 1129-5136, 2021.
[19] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization, 2018.
[20] Y. Wu, Tucker, and O. Nachum. Behavior regularized offline reinforcement learning, 2019.
[21] S. Ross, G. J. Gordon, and J. A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2011.
[22] D. Pomerleau. Alvinn: Proceedings (NeurIPS) Neural Information Processing Systems, pages 305 313. Morgan Kaufmann, December 1989.
An autonomous land vehicle in neural network. In Touretzky, editor,
[23] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning, 2020.
[24] N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. 2019.
[25] K. Kang, P. Gradu, Choi, M. Janner, C. Tomlin, and S. Levine. Lyapunov density models: Constraining distribution shift in learning-based control. In K. Chaudhuri, Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine I Learning Research, pages 10708 10733. PMLR, 17-23 Jul 2022. URL https::/ maleeeeddinn.m............ mlr, wwtppp/ewe//aaaeeppeeep html.
[26] M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning. 2022.
[27] Peters, K. Muelling, and Y. Altun. Relative entropy policy search. In Proceedings 24th AAAI Conference on Artificial Intelligence (AAAI '10), pages 1607 1612, July 2010.
[28] G. Chou, N. Ozay, and D. Berenson. Model error propagation via learned contraction metrics for safe feedback motion planning of unknown systems. In 2021 60th IEEE Conference on Decision and Control (CDC), Austin, TX, USA, December 14-17, 2021, pages 3576-3583, IEEE, 2021.
[29] C. Knuth, G. Chou, J. Reese, and J. Moore. Statistical safety and robustness guarantees for feedback motion planning of unknown underactuated stochastic systems. ICRA, 2023.
10

---

[30] G. Chou, N. Ozay, and D. Berenson. Safe output feedback motion planning from images via learned perception modules and contraction theory. In Algorithmic Foundations of Robotics Proceedings of the Fifteenth Workshop on the Algorithmic Foundations of Robotics, WAFR 2022, College Park, MD, USA, 22-24 June, 2022, volume 25 of Springer Proceedings Advanced Robotics, pages 349-367. Springer, 2022.
XV
[31] D. P. Kingma and M. Welling. Auto-encoding variational bayes, 2022.
[32] D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution, 2018.
[33] J. D. Co-Reyes, Y. Liu, A. Gupta, B. Eysenbach, Abbeel, and S. Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings, 2018.
[34] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks, 2014.
[35] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning, 2018.
[36] Ho and Ermon. Generative adversarial imitation learning, 2016.
[37] D. Rezende and S. Mohamed. Variational inference with normalizing flows. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pages 1530-1538, Lille, France, 07-09 Jul 2015. PMLR. URL https:/// h/pprc/eddnngg.....o mlr. www.:rpeesee/.eemm //7/s/////rreeed115 html.
[38] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and Ommer. High-resolution image synthesis with latent diffusion models, 2022.
[39] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
[40] Y. Song and Ermon. Generative modeling by estimating gradients of the data distribution, 2020.
[41] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2023.
[42] F. Permenter and C. Yuan. Interpreting and improving diffusion models using the euclidean distance function, 2023.
[43] Latorre. P. Rolland, and V. Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization, 2020.
[44] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2021.
[45] L. Manuelli, Y. Li, Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence model-based reinforcement learning, 2020.
[46] B. Trabucco, X. Geng, A. Kumar, and S. Levine. Design-bench: Benchmarks for data-driven offline model-based optimization, 2022.
[47] B. Trabucco, A. Kumar, X. Geng, and S. Levine. Conservative objective models for effective offline model-based optimization, 2021.
[48] M. Vlastelica, S. Blaes, C. Pinneri, and G. Martius. Risk-averse zero-order trajectory optimiza- tion. In Proceedings of the 5th Conference on Robot Learning, volume 164, pages 444 454. PMLR, 08-11 Nov 2022.

---

[49] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids, 2019.
[50] D. Kingma and J. Ba. Adam: method for stochastic optimization, 2017.
[51] P.-.. de Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein. tutorial the cross-entropy method. Annals of Operations Research, 134(1):19-67, Feb 2005. ISSN 1572-9338. doi: 10.1007/110479-0055572222
URL
[52] G. Williams, A. Aldrich, and E. Theodorou. Model predictive path integral control using covariance variable importance sampling, 2015.
[53] S. Ghadimi and G. Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming, 2013.
[54] H. J. T. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do differentiable simulators give better policy gradients?, 2022.
[55] C. Duchi, L. Bartlett, and M. Wainwright. Randomized smoothing for stochastic optimization, 2012.
[56] H. J. T. Suh, T. Pang, and R. Tedrake. Bundled gradients through contact via randomized smoothing, 2022.
[57] Marsden and M. Hoffman. Elementary Classical Analysis. 1974.
[58] Coles. An introduction to statistical modeling of extreme values. Springer Series in Statistics. Springer-Verlag London, 2001. ISBN 1-85233-459-2.
[59] P. Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):161-1674, 201 doi: 1o1:10.11 a 001422
[60] Li, X. Zhan, H. Xu, X. Zhu, J. Liu, and Y.-Q. Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference Learning Representations, 2023. URL https:/ wwwpereevew..eet/eeeeeeeeeeeeeee id=1M07TC7
[61] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision-making?. 2022.
[62] R. Tedrake. edu.
Underactuated Robotics. 2023. URL https://underactuutell. csail mit.
[63] O. Rybkin, C. Zhu, A. Nagabandi, K. Daniilidis, I. Mordatch, and S. Levine. Model-based reinforcement learning via latent-space collocation, 2021.
[64] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012. doi:10.1 R0...///....0...2.3...
[65] J. Richalet, A. Rault. J. Testud, and J. Papon. Model predictive heuristic control: Applications to industrial processes. Automatica, 14(5):413 1978. ISSN 0005- 1098. doi:https://doi. 0t//0/.0006600051101888899000010 URL htttps://ww..sciencellle httpe/iiietttttcceecooooomomm hte//science / articcle/pp./000511999990000
[66] A. Argenson and Dulac-Amold. Model-based offline planning, 2021.
[67] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets, 2021.
12

---

[68] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning, 2021.
[69] G. Chou and R. Tedrake. Synthesizing stable reduced-order visuomotor policies for nonlinear systems via sums-of-squares optimization. CoRR, abs/2304. 12405, 2023.
[70] C. Finn and Levine. Deep visual foresight for planning robot motion, 2017.
[71] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models, 2023.
[72] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong. I. Krasin,
D. Duong, A. Wahid, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation, 2022.
[73] Pang, H. T. Suh, L. Yang, and Tedrake. Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models, 2023.
[74] J. D. Lee, M. Simchowitz, M. I. Jordan, and B. Recht. Gradient descent only converges to minimizers. In Conference on learning theory, pages 1246-1257. PMLR, 2016.
[75] A. Braides. Caamaa-Cnvvrggece for Beginners. Oxford University Press, 07 2002. ISBN 9780198507844 doi: 1.. http:///000//0000000000000000000 10. 1w110099///// www.//////ooffff./
URL https://doi. org/
: oso /00.//9000880784000 001.0001.
[76] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
[77] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention MICCAI 2015 18th International Conference Munich, Germany, October 5 9, 2015, Proceedings, Part III, volume 9351 of Lecture Notes in Computer Science, pages 234-241. Springer, 2015.
[78] Y. Song and S. Ermon. Improved techniques for training score-based generative models. In NeurIPS 2020, 2020.
13

---

A Proofs
A.1 Proof for Proposition
For each k, we denote the stationary point of gradient descent as zi lim Tb. Using the first-order optimality condition at the stationary point, we know that
0 V vador D);
1==
Vn log
Σ exp
1or
xzl1?
20
x=x
(13)
of E, exp
1
Ok -2 (x:
20
Σi exp
|| - x;
20%
2=0
We then have that
Σ exp
|
22:12]
(1 xzi
exp
2n
1
||=
2all3]
)
Σ exp
2n
22|1?)
24
(14)
x
Σ
exp
2
10
x
112
21.
Σn exp
1 |
i
20
2.l12 and A,(o)) = - r;l12 - || 225122) Let 2j E arrmmnn 2EED |
Lemma 4. When there is a unique minimizer 2j, lim z1 = Zz.
Proof. Since zj is the unique closest data point to of. A > 0 for i # j and 4, 0. With the
monotonically decreasing de - 0, we know that
= f0 0
if =
lim A,(ox) k-000
(15)
oo
otherwise.
Therefore,
lim exp [-]], = { 80000
if
(16)
otherwise.
We then obtain
cxp
20 (11=g - 2:l12 11 - 22,11))
exp
1 A (Ok)
lim
lim k
1 (1|=%
xx.1? 11 22 12513)
Enjj exp
(ox))
+1
exp
20
(17)
{
if = otherwise.
Therefore, by combining (14) and (17), we have that
lim ot Tz. 10000
(18)
L
Lemma 5. When there multiple minimizers Si E argmin ||l xx
limp 2 21=1 Zt.
14

---

Proof. In contrast to (17) for the unique 2j, we have that
exp
(11=x x4112
|10 J/j1 [13)]
lim
10000
Σn exp
2 (1= - Fnll² 21 ||0 2. [13]]
exp (ox)]
lim
(19)
k-to0
Jm) exp
An (rx))] + m
1
if E fm
0
otherwise.
Therefore, 2 converges to the geometric center of all the minimizers:
lim
4k
Ij
(20)
1=1
Lemma 6. For a local minimizer zie 2z is unique: for a local maximizer or saddle point ot. z. is not unique.
Proof. We first prove that zj unique if 2 is local minimizer by contradiction. If there are multiple minimizers Tj.s. f E argmin || =z11?, we have that
lim do, (zt: 80000 k0000
lim - of log Σ
exp
20
1 | - xzll?)
o2
llog
lim
Σ exp
20 1 xz,1
2/
k-+00
(21)
min lim || xz ||
lim
| 21 11
#-000
Ty Kf
m
We observe that ot is local maximizer of min day (1;; D)2, which raises contradiction.
Similarly, we prove that z, is not unique a a local maximizer or saddle point by contradiction. Assume vy the unique minimizer of mina,e || 2x we have
lim da D)² = min lim || k0000
10000
lim k-000
|0
x2112
(22)
Hence, o can not be a local maximizer or saddle point of min, doy (z; D)?, which raises contradic- tion.
Since gradient descent converges a local minimizer almost surely with random initialization [74] and appropriate step sizes, there is unique x, for 2 and (18) holds. We also note that similar conclusion can be reached by using I'-oovvereence of the softmin to min or + [75].
15

---

A.2 Proof for Proposition 2
Let Le be the local Lipschitz constant of the enror e(x) := f(x) f(x))fo(2z) over some domain 2 S X, and define Ie == arg minz,eD 2 || x; i.e., the closest data-point. Then, we have the following:
[f(x) - fo(x))l 1 min [e(z;) + Lell 2z [ ,DED
S c(xx)) + Le 1e| rell2
=e(x)) + v2L.
|| zel1
= =e(xe) + -2L
min || xi11
(23)
=e(x)) + v
min FED
|| x2:12
<e(xe) + v2leg11 0² log Σ exp
|
x;ll²)
))
+ o² log N
20
=((x,) + V2L. v dz D)3
where C2 == log N C, and C is the constant defined in (4). In the second line, we use the fact that as Te is a feasible solution to the minimization in the first line, is an upper bound on the optimal value. In the sixth line, have used the fact that for any vector v [vi,..., ".] ]T E R". minfe1..... Un S log Σi_ i=1 exp( (tt;) + loga for some scaling In the final line, have applied the definition of (r; D) from (4).
A.3 Proof for Proposition 3
The perturbed data distribution can be written of Gaussians, since
Pa(x') = / Mo) o2Ddd
/ Σ ((2)) N(r'sa, T
1
Σ / ((:;)W
(24)
N
N(Inzz,01)
N
1
Σ
N(
N
16

---

Then we consider the negative log of the perturbed data distribution multiplied by o²
-o² Po -o log
Naz;zi,, I²
N
- log
Σ N(2;00 02)
+ log N
- log
Σ cxp
1
|> zil²²
+ + loolog log N
(2TT)T
20²
(25)
= o² log
exp
20 || - 2ill?
+ t log + o² log log(2To)
2
-o LogSumExp;
20 || - x:ll?
+ +CNN
Softmin,
x xil1? + nCN,, E)
where we define C(N, n. Σ) = 0² (log N + nn22 log(2
A.4 Function Error vs. Gradient Error
We illustrate with an example that in the finite error regime, bounded function error does not necessarily imply bounded gradient error.
Suppose we given a function f(x) IR + IR, and another function of the form
( f(x) + cos(war)
(26)
Then, the error between the two functions is bounded by
e(x) = f - g(x)ll < a cos(wz) S a for all x E IR. One might make a arbitrarily small (but not zero) to decrease the error.
(27)
However, consider the error in the gradients,
(x) [z) Vg(=)] aw sin(wx) S aw
(28)
which now scales with the frequency term w, which can be arbitrarily scaled up increase gradient егror.
B Details of the Planning algorithm
B.1 Computation of Gradients
Recall that the gradient of log Po (x;; u.) cost w.r.t. the input variable uy can be written as
Va, log, u4) = Vz, log p(x,, D., + Va, log p(z;, u. D,) u, where D denotes the Jacobian. Writing the dependence on each variable more explicitly, we have
(29)
V logp(z((u5). uu (u())) = V.. logp(z;(u;), Du, + V., log (og u, (ug)) u;())
(30)
(31)
where we note that Du, u, 1 = j and otherwise. As long as > j. we also note that d; has dependence
uy- Instead of computing this gradient explicitly, to compute (uy), u, (uy), and compute the score function. Then we ask: which quantity do need such that it gives the above expression when differentiated w.r.t. uz? We use the following quantity,
first rollout the trajectory
Gj = Sx (Ii u;) o; (uy) + (n,,, (u))( where the score terms have been detached from the computation graph. Note that c.j is scalar and allows us use reverse-mode automatic differentiation tools such pytorch [76].
(32)
17

---

B.2 Noise-Annealing During Optimization
Additionally, anneal the noise level during iterations of Adam. Given sequence og with K being the total number of annealing steps, we run Adam for max itermax / iterations, then run it with the next noise level.
B.3 Algorithm Block
We first give some abbreviations to simply the algorithm description. Given the dynamics, we write down the value function of Equation (11), conditioned in the noise level Ø,
T
V( = Σ ri (x2, u1) + =08
Σ log Pa (x2, ue: D)
(33)
s.t. J++1 fo ut)
Vt
We note that our implementation, use Adam [50] instead of doing gradient descent.
Algorithm Score- Guided Planning (Gradient Descent Version)
Require:
Learned dynamics fe. score function s(r, u), noise schedule sequence du- Initial guess u(.2- initial noise øp-
Require:
while not converged do
Rollout u².r and compute state-input trajectory (Izz, )e)
Compute Vanr (u ok) using score estimator s(x, u, ()) Equation (12), Appendix B.1
M1:T
k+1 &- ui.r - hV un. Vufier;
& Gradient Descent with stepsize
< le
end while
B.4 Connection to Diffuser
We first lift the dynamics constraint into quadratic penalty and write the penalty as log p( (rt, mz). This equivalence is seen by considering case where we fix tt, and perturb Te41 with a Gaussian noise of scale a. If (I,, ue 2++1) is in the dataset, it obeys Zz41 = f(z,, u,) under real-world dynamics f. This allows us to write
Po ( l u4) = ( 11 02)
log (I141 u.) - | - f(e,, =
(34)
where C is some constant that does not effect the objective. Then, we rewrite our objective using the factoring p(a,, u,) p(,;))))((;;) This allows to rewrite the objective of Equation (11)
T
Σ rT ue) + Σ log (gg((ul + B
log p(xx) + B Σ log p( ( u4)
t=1
(35)
=V(x1 upr) + logp(rr. uIr) +
log p(x),
1=1
where the first two terms are the objectives in Diffuser [26].
B.5 First-Order Policy Search
We note that our original method for gradient computation can easily be extended to the setting of feedback first-order policy search, where we define the uncertainty-penalize value function as
T
max
Exjop
Σ
relase (x) + =BO
log Pa (Xt, uz; D)
(36)
S.L. It+1 = fol us), ue = To (xt)
Vt E [1,7].
18

---

where is some distribution of initial conditions. We rewrite the objective with an explicit dependence on o, and use Monte-Carlo estimator for the gradient of the stochastic objective,
V auupp
F((ai(a),u(a)) + 3o²
(Ia(a,
-ppp V
(r(an(a),u + 3o²
Σ ra) ur (a)) + 30
Σ (o u s.t. 21 = 2i ~ p
(37)
where the last equation denotes that fixing the initial condition to sampled from p. and N is the number of samples in the Monte-Carlo process. Since r; and fo are differentiable, we can obtain the gradient
(38)
after rolling out the closed-loop system starting from I; and using automatic differentiation w.r.t. policy parameters a. To compute the gradient w.r.t. the score function, we similarly use the chain rule.
and differentiate it w.r. o, which lets compute
θ
Σ log (2 u1)
Σ
log Po (x)(a),
da
da
Oxt
Ou:
log Po (xt; ue)
+
log po (ft, u.)
(39)
Oz.
da
Oue
Oo
Oet
Our
8 (x1, un 0)
8n (rt: u4i 0)
da
da
where the last term is obtained by differentiating
Sa uei ue o) + (u(
(40)
after detaching 8a and s. from the computation graph.
B.6 Imitation Learning
We give more intuition for why maximizing the state-action action likelihood leads to imitation learning. If the empirical data comes from an expert demonstrator, maximizing data likelihood leads to minimization of cross entropy between the statc-action pairs encountered during planning and the state-action occupation measure of the demonstration policy, which estimated with the perturbed empirical distribution Po (2,, uz).
Loggo u.)
Σ log (uel2)) +
Σ log (x).
(41)
Note that the log po (x)) is identical to the Behavior Cloning (BC) objective, while log (x.) drives future states of the plan closer to states in the dataset. We note that Adversarial Inverse Reinforcement Learning (AIRL) [35] minimizes similar objective ours [13].
19

---

Experiment Details
C.1 Cartpole with Learned Dynamics
Environment.
We use the cart-pole dynamics model in [62, chapter 3.2], with the cost function
being
xlllo i
(x2,, ue)
(42)
else.
Q diag(1, We choose the planning horizon T 60.
Training. We randomly collected dataset of size N = 1, 000, 000 within the red box region in the state space. The dynamics model is an MLP with 3 hidden layers of width (64, 64, 32). For ensemble approach we use different dynamics models, all with the same network structures.
We train score function estimator, represented by an MLP with hidden layers of width 1024. The network is trained form 400 epochs with batch size of 2048.
Parameters During motion planning, for Adam optimizer we use learning rate of 0.01. For CEM approach we use a population size of 10, with standard variance a 0.05, and we take the top 4 seeds to update the mean the next iteration.
Data Distance Estimator. To train data distance estimator, introduce function approximator d, : R" RR+ + R parametrized by 7 predict the coise-dependent Softmin distance. The training objective is given by
1 min = Br BEEE, 2
d(z, a) Softmin,, ED || 2, 21
(43)
where Q is large enough domain that covers the data distribution D. For small datasets, is possible loop through all r; the dataset to compute this loss at every iteration. However, this training can get prohibitive all of the training set needs to be considered to compute the loss, preventing batch training out of the training set.
C.2 D4RL Dataset
We directly use the D4RL. dataset [44] Mujoco tasks [64] with 3 different envi- romments of halfcheetah, walker2d, and hopper. We additionally use differnet sources of data with random, medium, and medium-expert.
Environment.
Training. The dynamics and the score functions are both parametrized with MLP with 4 hidden layers of width 1024. The noise-conditioned score function is implemented by treating each level of noise as an integer token, that gets embedded into 1024 vector and gets multiplied with the output of each layer. This acts similar to a masking of the weights depending on the level of noise. The D4RL environment does not provide us with a differentiable reward function, so we additionally train an estimator for the reward. We empirically saw that for score function estimation, wide shallow networks performed better. We train both instances for 1000 iterations with Adam, with learning rate of le and batch size of 2048.
We additionally set a noise schedule to be a cosine schedule that anneals from a 0.2 to 0.01 for 10 steps in the normalized space of x, u.
Parameters We used range Bs between le and -1 depending on the environment, where in some cases it helped to be more reliant on reward, and in others it's desirable to rely on imitation. We use a MPC with T' = 5 and optimize it for 50 iterations with aggressive learning rate of -1
20

---

C.3 Pixel Single Integrator
Environment. In this environment, we have 2D single integrator (2), = 2t + ut I²2 Ue E I22 as the underlying ground-truth dynamics; however, instead of raw states It. we observe a 32 x 32 grayscale image = h((+)) E 1222322 which are top-down renderings of the robot. In these observations, the position of the robot is represented with a dot. Moreover, we assume that we do not directly assign the 2D control input u,, instead propose 32 32 grayscale image ut, where the value of the 2D control action U, is extracted from the image via spatial average:
U4
Σ (Pe:Pa)
gulPos Pay ue (Pa: Py):
(44)
where the sum loops over each pixel (pz, Pоy {1,.... 32)2 yr (Px. Py) refers the intensity of the image pixel (Pz+Py) Py), and Iu x Z - I² is grid function mapping from pixel (Pa.Py) to corresponding control action. The control image ú, is normalized such that its overall intensity sums to 1. Given some goal Zg. the reward is set to be the -ce(ze, u4), where the cost (z uz) is
||
Fg Q. | |
if T
ca(2,, un)
(45)
||,,
Fgg 1.
else,
To evaluate this cost function for the planned sequence of image observations and control images, the states I, are also extracted from the image observations through similar spatial averaging:
Tt
Σ (Pa)Pa)
9im (pm Pyy Pyy Py);
(46)
where fim : - x + R2 is a grid function mapping from pixel (pa, Py) to a corresponding state.
In other words, we have running costs for the state and input, and a different terminal cost for the state. We set R 6.51, Q 500I, and Qa 1000I, and plan with horizon of T 15.
200, 000, with underlying 2D data sampled from I2 [-1, 1]2 and uy [-0.2, 0.2]2. Both the dynamics and the score function estimator are represented as U-Nets [77], with the architecture coming from [78].
Training. We collect randomly collected dataset of size
Parameters. We found that 3 0.5 is sufficient for the penalty parameter. Results obtained within 1450 iterations with learning rate of 0.03. In representing the noise-conditioned score function, we use 232 smoothing parameters { k= from 01 = 50 to d232 0.01.
232
Baselines. For gradient-based planning with ensembles, we set 3 0.5 and use an ensemble of size 10. For CEM with ensembles, we set 3 = 0.5, with ensemble of size 5 (we only used the first five networks in the original ensemble of size 10 due to RAM limitations).
Method SGP (Ours)
Cost of plan
Actual achieved cost
87.61
124.72
Vanilla MBRL
0.80
1 1862.06
Ensembles
4.31.54
1380.41
CEM
2244.72
2901.95
Table 2: Costs for pixel-space single integrator example (lower is better).
Numerical results. In Table 2, we report numerical results on the costs achieved by our approach and the various baselines tested the pixel-space single integrator example (Fig. in the main text). SGP (our method, Method A in Fig. 3) achieves low cost at planning time (87), and the actual achieved cost when rolling out the controls open-loop on the true system only slightly worse (124), since the model error is kept small. In contrast, for vanilla MBRL (Method B in Fig. 3 in the main text), overly-optimiset cost is achieved (0.8) at planning time, due to the generation of unrealistic images that exploit subtleties in how the cost is calculated; however, when executing the resulting
21

---

controls, the cost greatly degrades (1862) due to the model mismatch. Using ensembles (Method C in Fig. 3 in the main text) leads to the optimizer failing to find good control sequence at planning time, due to the poor optimization landscape when planning with ensembles (cost is 4031); this translates Fig.
to high cost when rolling out on the true system as well (cost is 1380). CEM (Method in the main text) similarly does not find a good control sequence at planning time, due the high dimensionality of the search space (cost of 2244); the actual cost achieved is similarly high (cost of 2901).
C.4 Box Pushing with Marker Dynamics
Environment. We prepare a box box- pox-piuhing environment where we assume that the box follows quasistatic dynamics, which allows us to treat the marker positions directly as state of the box that is bijective with its pose. We use 2D coordinates for each markers, and append the pusher position, also 2D, resulting 24 E IR 12 The pusher given relative position command with a relatively large step size [73]. In addition, we give the robot knowledge of the pusher dynamics, 2t+1 2t The general goal of the task is to push the box and align the edge of the box with the blue tape line.
pusher ut-
We formulate our cost as
Iaruarkerr
marker
if = T else,
ce ca Uz) =
QT
(47)
where Q7 = I and R = 0. In order to get mararrerr we place the box where we want the goal to be and measure the position of the markers.
Training. We collect 100 demonstration trajectories resulting in 750 pairs of (xt, ue Te+1). ). The dynamics and the score functions are learned with MLP of hidden layers with size 1024, with the noise-conditioned score estimator being trained similar to the D4RL dataset with multiplicative token embeddings. We train for 500 iterations with a batch size of 32.
We additionally noise schedule be cosine schedule that anneals from 0.2 to 0.01 for 10 steps.
Parameters. We observed that 3 = 1e performs well for all the examples, with learning rate of 0.1 and 50 iterations. We found that horizon of T = 4 was sufficient for our setup.
Numerical Results We report numerical results for the box pushing demo in Table C.4, which illustrates that SGP achieves muchy lower cost compared to vanilla MBRL.
Goal
Method
Achieved Cost
Left
SGP (Ours)
8,86 + 4.54
Left
Vanilla MBRL
51.18 + 6.93
Center
SGP (Ours)
7.74 2.23
Center
Vanilla MBRL
35.93 1.18
Right
SGP (Ours)
9.05 1: 3.45
Right Vanilla MBRI.
61.40 3.82
Table 3: Cost for the Keypoints hardware Example (lower is better). We compare the results of
running MPC using vanilla MBRL, vs. SGP. The cost is evaluated on the final achieved trajectory hardware. Numbers are averaged on trials.
22

---

D
Experiments on Hyperparameters on SGP
D.1 Effect Penalty term 3.
Note that 3 appears in the objective trade off the reward signal and the uncertainty penalty term,
T
T
max MT-T
Σ relar ub) + _08
Σ
log Po (2e, (0,, u4: D)
(48)
1=1
s.t. 1441 = fo( u4)
VI [1.
We first give intuition for two extreme cases.
D.1.1 Vanilla Model-Based Reinforcement Learning (MBRL), / =
When 3 is 0, no uncertainty is penalized, and the planning problem becomes equivalent to vanilla Model-Based Reinforcement Learning (MBRL), or planning with learned dynamics. While this achieves best performance in terms of fa. the optimizer is likely to not perform well under the true dynamics due to model bias.
D.1.2 Imitation Learning/ Data Landing, 3 ac
When 3 = oc, the reward term discarded and the problem simply turns to that of uncertainty minimization. When data comes from random sources, the problem will then plan a trajectory to stay near the vicinity the data depending on the value of Interestingly, the for when B comes from an expert demonstrator. We can write down
be understood case of imitation learning if data
T
T
log Po u:
(L u)ll((( u4)
(49)
where p is the occupancy measure of the plan, and thus maximizing this quantity leads to a form of distribution matching between this occupancy measure and the perturbed empirical distribution of data. We have more details in Appendix B.6.
D.1.3 Experimental Validation
To illustrate the effects of B, we prepare a single-integrator environment where the dynamics obey -t-1 = Ft. + ue, except in the circular region where actuation gets lost and Fe+1 Tt. For instance, this could simulate a pit in a self-driving environment. The effect setting different (3s are illustrated in Fig.5.
As hypothesized, too high of 3 results in not making progress due to no reward signal the cost is quite high. Yet, the dynamics error is very low. On the other hand, too low of 9 results in a big dynamics error as the single integrator goes into the middle pit region and makes no progress. As result, the overall cost is very high as well.
result,
We note that around -2 and 3 le the sweet spot for this experiment, where the agent was able circumnavigate the pit as it has not data there, and still make meaningful progress towards the goal.
D.1.4 Tuning Recommendation
3 to 8 = 1e and explore performance In practice recommend setting between values of B -3 tradeoffs by doing hyperparameter search.
23

---

B le
B le
2
B 1e
β 1e
524
Hallucinated
0.75
start
6.50
-10
-25
3.0
0.5
1.3
0.75
12811
0.75
0.75
0.75
0.50
3.0
0.50
0.50
Real
0.25
0.20
9.25
0.20
0.00
3.0
0.00
0.00
-1.2
350
4.50
-0
3.79
-0.75
-14
-05
5.0
0.5
1.0
-1.
-0.5
3.0
55
-30
-0.5
Beta vs. True Cost
Beta vs. Dynamics Error
14
Dynamics Error
Cost
8-
6
4-
0-
0.0
0
2
(log scale)
Beta (log scale)
Figure 5: Top row: The hallucinated trajectory with different choices of B. Middle row: real rollouts with different choices of Bottom row, left: sweep of B in log-scale and cost. Bottom row, right: sweep of B and the dynamics error on the rolled out trajectory.
D.2 Effect of Noise Level o.
Similar to 3, the performance of the algorithm deteriorates under too high too low of a value for the injected noise a.
D.2.1
Effect of low a
The biggest factor preventing us from having too low of a a comes from the learning the score function. denoising score matching works by perturbing the data with the given noise a, only regions within a small vicinity of the data will have been covered by score matching regions has been trained on, the
low.
As the training accuracy of neural networks deteriorates accuracy of score matching suffers greatly with too low of This effect is extensively illustrated by previous score matching papers such as [40].
Another effect is that without normalization by a, the log probabilities the perturbed empirical distribution has scaling that directly depends on a. This clear to see with single data, for which the perturbed empirical distribution becomes Gaussian.
-logp z
202 +
(50)
This suggests that becomes too small, the Lipschitz constant of the distribution becomes large, creating landscape that is challenging for neural networks to learn.

---

To visually validate this, we illustrate the efficacy of score matching as we vary a in Fig.6. As we hypothesize, score matching is very ineffective for small a, and the actual gradients are very jagged / near-discontinuous in this regime. On the other hand, as we increase o to around 0.1 and 0.2 level, score matching successfully predicts gradients log-likelihood. However, even for = 0.2, note that we cannot predict the score accurately if we get too far away from the data, since the score estimator never sees these points during training.
o 0.01
= 0.05
o
Pertubeed Data Detribtion
Astediff Vc. Score Matchhing
Forturbed Ortribution
Score Hatching
AutiCff Nascling
12-
Score Naich-1g
25
12
5030
290
Netubbed Bution
Artuded Datadan
2-
2
-20090
-400
0.1
0.2
0
Pertureed Data Dimrbuuin
Astodif Vs. Scon-Matchnn
Perturbed Data Detribtionn
Autodif vs. Scone-Hatchig
20-
Scee Maiching
12-
Osta
Carnelatiing
12-
15-
68-
1.-
-300
Figure 6: Efficacy of Score Matching vary the perturbed noise a. note that this is in the unnormalized space.
D.2.2 Effect of high o
If a is too high compared to the scale of how close data are together, then it is hard to distinguish from a pure Gaussian noise centered around the average of the data. Indeed, this is what the "nosing process" of diffusion models rely on.
As the goal of SGP is stay near the data, not go towards the average of data, it not desirable to choose too high of In fact, provide an adversarial case in the pit example, where we have not collected any data near the mean of the data. In such cases, too high of a can lead to catastrophic failure.
D.2.3 Experimental Validation
We validate our choice of a by training a score estimator for the example in Appendix D. and visualizing its behavior. Our experiments validate our hypothesis that when function is very inaccurate. On the other hand, too big of leads the score function to point towards the center of all the data - which in this case, corresponds to region where data was not collected. In between we have a sweet spot where the score function pushes away from regions with no data.
small, the score
D.2.4 Tuning Recommendation
To whether user has chosen good a, recommend that the user first tests the validity of the score estimator before attempting to solve any optimal control problem. This can be done by training score estimator, and running gradient descent on the log probabilities using the score estimator as
25

---

0.01
0.1
0.3
= 0.5
(N=100 (N100,000) 10,00
8
Dense
1.0
-10
-1.0
3.0
Sparse (N=300)
60
0.0
-1.0
Figure 7: Visualization of the learned score function s(x) V log (x)), according to different a values. Top corresponds to a dense and bottom corresponds to a sparse dataset. The grey region
corresponds to region with no data. Note that corresponds to noise in the normalized space.
done in [40]. A good score estimator should ensure that after some iterations, the iterates converge to samples within the data.
We note that in the normalized space, we often find 0.05 0.3 to good. To tradeoff between accuracy and coverage within this range, anneal in between this range following [40].
E Experiments on Extrapolation
One limitation of SGP that was mentioned was that the method does not have the ability to extrapolate.
We show that,
1. On the trajectory level, our method can successfully extrapolate by stitching trajectories to create a trajectory that was not in the dataset.
2. Not doing extrapolation can very practical method on pathological cases where extrapo- lation actually hurts.
E.1 Trajectory-level Extrapolation by Stitching
In order to show trajectory-level extrapolation capabilities, replicate the example in [26] by preparing types of demonstration trajectories. The first type of demonstration goes from the bottom left to right top, while the second type of demonstration goes from top left to bottom right. Then, SGP is asked to create a trajectory that goes from bottom left to bottom right- as this kind of demonstration has never been seen before, this requires use to extrapolate in the space of trajectories and perform a type of trajectory stitching.
As results in Figure 8 indicates, SGP is successful in stitching sub-trajectories from the two demonstrations to come up with entirely new trajectory that was not in the demonstration set. Thus, our method is capable of performing extrapolation in the space of trajectories.
E.2 Dynamics-Level Extrapolation as double-edged sword
By actively constraining the state-action pairs to stay close to the dataset, SGP discourages dynamics- level extrapolation. While this can be a possible limitation, we also show that this is an important strength of our method compared methods that attempt extrapolate.
For this purpose, we consider the single-integrator with a data-hole inside, where the agent has not seen data before. The true dynamics is defined
[1 We if outside hole inside hole
F++
(51)
26

---

Demonstration, Type
Demonstration, Type
10
1.0
0.5
0.5
car
0.0
0.0
-0.
-1.0
-10
-1.0 -0.5
0.0
0.5 1.0
-1.0 -2.5
0.0
0.5
1.0
Figure 8: Left: Demonstration trajectories that go from left hottom to right top. Center: demonstration trajectories that go from left top to right bottom. Right: performance SGP when asked to find trajectory from left bottom to right bottom, which was not in the dataset before.
These scenarios are plausible if the hole region falls outside the normal operating conditions of the agent. For example, self-driving car might always be forced go around a pit, or patches of grass in rotary. Naturally the dynamics still defined in the physical world, but the agent has never seen data there. We test the performance of SGP vs. Ensembles in this scenario and plot the results in Figure
As the dynamics are very simple outside the hole, ensembles will learn the simple dynamies F++11 Tr + U, with very high-confidence. As result, the trajectory found by penalizing ensemble variance attempts extrapolate, cutting across the data hole region where it believes the dynamics is still single integrator. Yet; the extrapolation fails and the method gets stuck at making progress.
However, SGP avoids this problem by making no assumptions about the extrapolation region, and simply avoiding it. This allows it to escape the pit region and still succeed in reaching the goal.
This experiment goes show that attempting extrapolation is doublc-edged sword. While the agent may have learned a generalizable form of dynamics that allows it to perform well, it also may not have, and there is no data to test it. Thus, in cases where we do not have enough domain knowledge or inductive bias in the network, it can be a strength to avoid extrapolation rather than limitation.
SGP (Ours)
Ensembles B le5
B = le
goal start
start
Hallucinated
050
0.00
-0.0
0.50
-0.75
-0.35
-10
-0.
0/5
0.5
1.0
-1.0
-0.
0.0
0.5
1.0
-W- goal
-#- Start
0.75
0.50
Real
0.55
0.23
-0.25
-025
-0.75
-0.75
3.0
Figure 9: Comparison of SGP vs. Ensembles variance penalty in the single integrator with data hole example. While SGP successfully avoids the hole, ensembles overconfidently extrapolate and fail to plan good trajectory that transfers to real-world dynamics.
27


Reviews:
[{"review_id": "rf-3fKTazHf", "reviewer": "Reviewer_bqCh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes modelling the score function of the perturbed empirical data distribution in order to capture \"guide\" the optimization closer to the data, which is a way of tackling uncertainty in planning methods. Further as a theoreticall result i is shown that the NLL of a perturbed empirical distribution is equivalent to a smoothed distance to the data up to a factor. Results are provided in simulated domains and on real hardware.", "summary_of_recommendation": "there are two reasons for this rating:\n1) I doubt that this method is practical, since there is no extrapolation beyond data (could be convinced otherwise)\n2) unclarities (see review)", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "line 97, take a look at [1] and cite, since they deal with exactly what you are describing (uncertainty penalty, aleatoric/epistemic separation, ensembles), PETS doesn't take into account uncertainty.\n\nproposition 1 - this statement is almost trivial and well-known? Why is it necessary to state it here formally? And also to give a proof?\n\nproposition 2 - in which sense is this Lischitz constant \"local\"? How do you define this locality? Also is it defined in the classical sense $\\| x - y \\| \\le L \\| f(x) - f(y) \\|$? Details seem to be missing here.\n\n\nfor the pairwise finite slopes estimation of the Lipschitz constant, if I understand correctly, it doesn't seem to be \"local\". Can you elaborate more on the estimation of the constant?\n\ndefinition 3.2 - isn't this just a mixture of Gaussians with equal weghts and variances where the means are centered at the data?\n\nline 156 - in which sense there are no guarantees on the quality o the gradients of the likelihood function? Can you clarify this statement? If I am able to estimate the true log-likelihood, then I have the true grad log p, what does \"quality of gradients\" mean in this context?\n\neq. 9 - so you are actually modelling the mixture of Gaussians  with the smoothing parameter $\\sigma$ then you start increasing it? Shouldn't the sampling distribution of x' at some point have 0 mean and variance depending on the variance schedule? As per your definition, the sampling distribution of x' is always centered on x.\n\nline 175 - I think an algorithm box would be useful here, it is easy to glide over the fact that this method is only useful in combination with a gradient of the return. Moreover, it is not clear from the writeup how do you combine the gradients and what this corresponds to - is it just a heuristical addition with a hyperparameter? Why is this a good quality gradient that is resulting from this computation? This is a very important part of your method that is not explained well.\n\n\nline 178 - \"projected into space of decision variables\", I don't undertand this part of the sentence. Isn't the x that you talk about in the previous section equivalent to (x,u) here? So you are modelling the perturbed joint distribution ofx,u\n\n\ntable 1, experiments - you method tries to stay close to the data distribution, therefore perhaps a more suitable offline algorithm for comparison would be AWAC\n\nline 238 - subtleties*\n\nconclusion - you state that you have not investigated aleatoric uncertainty, however, your epistemic uncertainty estimate is also questionable, since you don't take into account model extrapolation, only distance to the data. This means that your model can perfectly extrapolate, but your method would penalize it nevertheless. This I believe is the advantage of ensemble methods.\n\n[1]  Vlastelica, Marin, Sebastian Blaes, Cristina Pinneri, and Georg Martius. \"Risk-averse zero-order trajectory optimization.\" In 5th Annual Conference on Robot Learning. 2021.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## clarity\n\nthe paper is mostly written clearly.\n\n## strengths\n\nthe work is well motivated\n\ntheoretical result connecting the smooth distance to data and the proposed score estimation\n\nrobotics experiments\n\n## weaknesses\n\nthe way that uncertainty is treated here, it doesn;'t extrapolate.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "dIVZ4DG0Juh", "reviewer": "Reviewer_Esgu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Offline RL needs careful consideration of uncertainty estimation. This paper examines the use of smoothed distance to data as a metric, demonstrating its ability to stably converge to data and facilitate analysis of model bias through Lipschitz constants. Additionally, the authors establish an equivalence between smoothed distance to data and data likelihood, enabling the application of score-matching techniques to learn distance gradients to data. The authors highlight that offline model-based policy search problems that aim to maximize data likelihood do not require likelihood values but solely rely on the log-likelihood gradient (the score function). To tackle high-dimensional problems where zeroth-order methods struggle to scale and ensembles fail to overcome local minima, the authors propose Score-Guided Planning (SGP). This offline RL planning algorithm leverages score-matching to enable first-order planning. Empirical results show comparable performance with baselines.", "summary_of_recommendation": "The paper is carefully written with robotics experiments. Source code provided. I choose to accept the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- In Table 1, why other methods do not have the confidence interval?\n- In Pixel Single Integrator, Box Pushing with Marker Dynamics tasks, what is the scale of \\beta tried after you find the best one?\n- Why BO, MOPO, CQL are not discussed in the related works?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-written. Technical novelty is sound, with highly relevant results to robotics. The work is interesting.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "NZ2sfFEbV4", "reviewer": "Reviewer_X384", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for offline model-based RL (MBRL) called score-guided planning (SGP). The method is introduced by suggesting that the distance to data is good uncertainty metric and that a smoothed version of this distance is just the data likelihood. The method then uses the score function of the data generating distribution as a regularizer to prevent a gradient-based (single shooting) planner from choosing OOD actions. Experiments are presented in simple simulated control problems, a single integrator with a pixel-based action space, and a real robot pushing task.", "summary_of_recommendation": "Overall, this paper presented an interesting idea and backed it up with a diverse set of experiments, including on robotic tasks. There is room for improvement, but I think the paper is a good contribution and should be accepted.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See weaknesses above.\n\nAlso, the paper explains how diffuser can be seen as a relaxation of SGP. But, what (if any) are the key differences between SGP and diffuser?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\n1. The core idea of using the score function as a regularizer for offline model based RL is interesting and provides a nice connection between generative modeling and offline RL.\n\n2. The experiments demonstrate qualitatively that the method is able to work effectively with high dimensional observation and action spaces.\n\n3. The examples about the potential weaknesses of ensemble-based uncertainty provide a nice illustration of why something like distance to data may be preferred in practical scenarios.\n\n## Weaknesses\n\n1. It would be good to present some more quantitative results in the experiments section. The qualitative results for the non-D4RL experiments are useful, but do not make clear quantitative comparisons between methods. \n\n2. The performance on the D4RL benchmark suite is not particularly strong. SGP gets some small wins on some D4RL datasets, but does so poorly on others that the average performance is worse than the baselines. It could also be interesting to see comparisons on some of the higher dimensional robotic manipulation tasks in the D4RL suite instead of just the mujoco locomotion tasks. ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "DOFDRZsg7A_", "reviewer": "Reviewer_9mZ5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper the authors (i) propose an epistemic uncertainty metric for learned dynamics models, namely smoothed distance to data, (ii) identify a connection to score-based generative modeling that enables tractable gradient-based optimization with objectives incorporating this metric, and (iii) apply this methodology in the context of offline model-based reinforcement learning in a novel algorithm called Score-Guided Planning (SGP). Because the uncertainty metric is amenable to gradient-based optimization, SGP scales to problems with high dimensional state/observation spaces (e.g., images) where zeroth-order optimization methods struggle. Experiments demonstrate that SGP outperforms other baseline uncertainty estimators and optimizers (e.g., ensembles, CEM) on standard benchmarks, and moreover succeeds in high-dimensional and data-sparse regimes where it also makes more accurate policy rollout predictions than vanilla (unregularized) MBRL.", "summary_of_recommendation": "I recommend this paper's acceptance at CORL 2023 as the offline RL problems it addresses (particularly those with high-dimensional perceptual input) as well as the methodology it develops are highly topical in the robot learning community today. As noted above, I believe that some of its claimed insights could be better explained/better probed by additional experiments, but such additions would only be supplementary to the existing sound core work.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- How were the range of $\\sigma$ values (noted in the Appendix, typically ending at 0.01) selected in practice, and what happens if you take $\\sigma$ too low (e.g., do you have to compensate by lowering $\\beta$)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The reinterpretation of score-matching for a perturbed empirical distribution as learning gradients of smoothed distance to data is a clever idea, and the paper is well-written with convincing experiments supporting the authors' main claims. I am pleased to recommend its acceptance, however I believe that there are two areas of improvement which, if addressed, would significantly improve the impact of this paper.\n\n1. *Relevance of the theory:* Section 3 argues that smoothed distance to data is a good choice for gradient-based optimization as it can be guaranteed that, when solely optimizing for this metric, gradient descent will converge to a point in the dataset as the smoothing parameter is taken to 0. This fact is not particularly surprising (e.g., an alternate/more constructive proof could reason about Voronoi cells, minus a shrinking boundary as $\\sigma \\rightarrow 0$, as regions of attraction), but perhaps critically it overlooks the possibility that selecting a spot between data points may be desirable. A common rationale to use neural networks to fit dynamics is, if not to generalize/extrapolate (e.g., through meaningful learned representations as mentioned in the discussion of limitations), at least to interpolate. In light of this, $\\sigma$ may be interpreted not only as a penalty for straying outside of the training dataset, but also as a factor allowing for interpolation. That is, $\\sigma$ seems like one of the more critical tuning parameters of this method (similar to how the choice of a kernel bandwidth parameter is regarded as a key tuning parameter for Gaussian process-based methods), and may deserve more empirical study than the theory might suggest.\n\n2. *Experimental evaluation:* As noted above, the experiments in this paper do convincingly demonstrate the applicability of smoothed distance to data to gradient-based trajectory optimization. Beyond that, however, the experiments do not yield much insight on how SGP's performance depends on the choice of its tuning parameters (i.e., $\\sigma$ and $\\beta$). For example, in line with the previous point, studying sparser data (i.e., lower $N$) or finer time discretization may be a way to study the role of $\\sigma$ in allowing for interpolation. Also, arguably all of the experiments are conducted on effectively low-dimensional state/action spaces (by construction in the case of the Pixel-Based Single Integrator). What happens if you try the real-world box-pushing task with image observations instead of mocap marker positions? Though undoubtedly mechanically SGP can perform gradient-based optimization in multi-thousand-dimensional spaces, if the method doesn't scale to tasks where you cannot ensure that a solution trajectory can be assembled as a sequence of states you've seen in the training dataset (i.e., *data stability* is impractical, which would at face value seem to be the case for increasingly high dimensional systems), it may be worth mentioning in the limitations section (e.g., this may be less of an issue for uncertainty quantification methods that attempt to infer model confidence irrespective of distance to data in the input space).\n\nMinor comments:\n- In the problem definition of offline model-based RL which mentions an empirical distribution over state transitions (line 93), it initially appears that the dynamics themselves may be stochastic (i.e., contain aleatoric uncertainty). It is mentioned only later in the last line of the limitations that aleatoric uncertainty is uninvestigated; for reader clarity it may be worth mentioning earlier (i.e., in Sections 1 and 2) that the dynamics are assumed deterministic and that learning $f$ can justifiably be treated as a (uncertainty-regularized) regression problem.\n- How do local Lipschitz constants (computable perhaps for the cart-pole system) of the learned dynamics $f$ vary as uncertainty is increasingly penalized?\n- Is some degree of data preprocessing (e.g., whitening) necessary to make squared (Euclidean) distance the right distance to penalize?", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper does model-based offline RL with a score matching regularization term that encourages the resulting trajectories to go towards the offline data. All reviewers have voted for acceptance after the authors' rebuttal. While the method is interesting, one of the main questions that has remained unanswered is why are the D4RL results not always competitive with existing imitation learning or offline RL methods? Also, one of the main advantages of using offline RL is to leverage the "stitching" behavior (available data from A -> B, B->C means that through offline RL the policy can figure out A->C even if it is not in the data). The loss function here forces the trajectories to stay close to the data, so it is possible that that hinders stitching behavior, which makes me wonder why is offline RL the right RL setting to showcase this method, i.e. why not online RL + the regularization term? Finally, I would definitely have expected that the paper cite and compare against COMBO https://arxiv.org/abs/2102.08363, which as far as I can tell it doesn't.

So, while I think I would personally argue for a rejection, as I don't think offline RL is the right setting for this contribution, I will respect the reviewers evaluations and recommend acceptance.
Decision: Accept (Poster)