One Model to Drift Them All: Physics-Informed Conditional Diffusion Model for Driving at the Limits
Authors: Franck Djeumou
Thomas Jonathan Lew
NAN DING
Michael Thompson
Makoto Suminaka
Marcus Greiff
John Subosits
Venue: CoRL 2024 (2024)
Keywords: Diffusion Models
Learning for Control
Autonomous Drifting
Model Predictive Control
TL;DR: A single diffusion model can adapt online to reliably control a Toyota Supra and a Lexus LC 500 on challenging autonomous drifting tasks

Abstract: Enabling autonomous vehicles to reliably operate at the limits of handling— where tire forces are saturated — would improve their safety, particularly in scenarios like emergency obstacle avoidance or adverse weather conditions.
However, unlocking this capability is challenging due to the task's dynamic nature and the high sensitivity to uncertain multimodal properties of the road, vehicle, and their dynamic interactions.
Motivated by these challenges, we propose a framework to learn a conditional diffusion model for high-performance vehicle control using an unlabelled multimodal trajectory dataset.
We design the diffusion model to capture the distribution of parameters of a physics-informed data-driven dynamics model.
By conditioning the generation process on online measurements, we integrate the diffusion model into a real-time model predictive control framework for driving at the limits, and show that it can adapt on the fly to a given vehicle and environment.
Extensive experiments on a Toyota Supra and a Lexus LC 500 show that a single diffusion model enables reliable autonomous drifting on both vehicles when operating with different tires in varying road conditions.
The model matches the performance of task-specific expert models while outperforming them in generalization to unseen conditions, paving the way towards a general, reliable method for autonomous driving at the limits of handling.

One Model to Drift Them All: Physics-Informed Conditional Diffusion Model for Driving at the Limits
Franck Djeumou 122 Thomas Lew', Nan Ding', Michael Thompson', Makoto Suminaka', Marcus Greiff', and John Subosits' Toyota Research Institute, "Rensselaer Polytechnic Institute
Abstract: Enabling autonomous vehicles reliably operate at the limits of han- dling where tire forces are saturated would improve their safety, particularly in scenarios like emergency obstacle avoidance or adverse weather conditions. However, unlocking this capability is challenging due to the task's dynamic na- ture and the high sensitivity to uncertain properties of the road, vehicle, and their dynamic interactions. Motivated by these challenges, propose framework to Iearn a conditional diffusion model for nigh-performance vehicle control using an unlabelled dataset containing trajectories from distinct vehicles in different environ- ments. We design the diffusion model to capture the complex dataset's s trajectory distribution through multimodal distribution of parameters of physics-informed data-driven dynamics model. By conditioning the generation process on online measurements, we integrate the diffusion model into real-tima model predictive control framework for driving at the limits, and show that it can adapt on the fly to given vehicle and environment. Extensive experiments a Lexus LC 500 show that a single diffusion model enables reliable autonomous drifting both vehicles when operating with different tires in varying road condi- tions. The model matches the performance of task-specific expert models while outperforming them in generalization to unseen conditions, paving the way towards a general, reliable method for autonomous driving at the limits of handling.
Toyota Supra and
Keywords: Diffusion Models, Learning for Control, Autonomous Drifting.
Online Trajectory data
Conditional Diffusion Model
Hz
Neural SDE de (a. 2 (x. wjdW
MPC 200 Hz
Figure I: Left: Examples of the conditional diffusion model performing drifting trajectories on two vehicles. Right: Overview of the controller architecture and online model parameter generation process. The videos of the experiments can be found at https: /tt:://tiiiurir www.coo////oo// diff-drift.
Introduction
Existing autonomous vehicles constrained operate fraction of their full handling potential. Designing algorithms to reliably control vehicles beyond these engineered limits would unlock faster
Conference on Robot Learning (CoRI. 2024), Munich, Germany.

---

and more reliable responses to diverse safety-critical situations [1, 2] such as driving on ice and avoiding sudden obstacles, scenarios where the vehicle may use all of the available tire-road friction, causing it to slide across the road [3, 4, 5]. However, driving at the limits of handling is challenging due to the task's dynamic nature, the high sensitivity to model mismatch, and the uncertain properties of the road, the vehicle, and their dynamic interactions. In addition, the high cost of collecting a dataset for driving at the limits, the complex vehicle dynamics, and the safety considerations complicate the use of imitation learning and reinforcement learning strategies. These challenges motivate the development of a model capable exploiting physies knowledge and capturing complex forms of uncertainties while being amenable to real-time autonomous vehicle control.
Diffusion models [6. 7, 9] have shown to be highly capable of representing complex, high- dimensional, and multimodal distributions from data. However, their direct use for driving at the limits is not straightforward. The limitations of classical diffusion models include the question of how to leverage prior physics knowledge to improve data efficiency and interpretability, and the considerable model inference time, which can be a bottleneck for high-bandwidth control.
Contribution. We propose a conditional diffusion vehicle model for control-oriented modeling of driving at the limits of handling under uncertainties. By predicting the parameters of physics- informed neural stochastic differential equation dynamics model, the model has four key properties.
By encoding prior physics knowledge as inductive bias, the proposed diffusion-based vehicle model is interpretable and generalizes to new environments from small amounts of data.
The model can capture complex multimodal distributions over the vehicle model's 's parameters.
The model can adapt the fly to various test-time vehicles and road conditions by conditioning on measurements of the vehicle's interaction with the world.
By predicting the parameters of physics- based model, as opposed to directly predicting state trajectories, the model inference and control loops are decoupled. This hicrarchical approach unlocks diffusion sampling at low rates and high-frequency predictive control.
We integrate the diffusion model in a real-time nonlinear model predictive control framework for autonomous driving at the friction limits, and extensively validate it on Toyota Supra and a Lexus LC500. Our results showcase that a single diffusion model can reliably control both vehicles on challenging drifting tasks involving different road conditions and vehicle properties, see Figure 1.
2 Related work
Several works have explored autonomous driving at the limits of handling, both in the context
of racing [10, 11, 12, 13] and drifting [14, 5, 4, 15]. These methods identify the parameters of a physics-based vehicle dynamics model [16, 17, 4, 18, 19] or train a neural network model [15, 20], and subsequently use for model-based optimal control. Nonlinear model predictive control (MPC) is the go-to control strategy in such settings and has demonstrated high- rege-performance tracking ability in challenging racing and drifting tasks [12, 11, 4, 15]. However, the performance of MPC is limited by the fidelity of the vehicle model, designed to capture a single vehicle with given tires operating in specific road conditions. In contrast, using an unlabeled trajectory dataset, we train a single generative vehicle model with online adaptation capabilities that enable autonomous driving at the limits of handling on different vehicles in varying road conditions.
Diffusion models have emerged as powerful tool for generating complex and multimodal distri- butions in continuous domains such as images [21, 22], 3D contents [23, 24), planning and control [25, 26, 27, 28, 29, 30, 31], time series [32, 33], and physics processes [34, 35, 36). However, all these works learn to represent, in a black-box manner, distributions for which samples are directly available in the training data. In contrast, we train diffusion model to generate samples from latent space of vehicle dynamics parameters that are not in the training data. Our approach relates to research in latent diffusion models [23, 37]. But instead of learning encoder and decoder net- works to map between the data and latent spaces, we impose a structure on the latent space through physics-infomme informed neural stochastic differential equations.
Neural ordinary differential equations [38, 39, 40, 42], Koopman operators [43, 44, 45, 46, 47, 48), classical system identification [49, 50, 51, 53]. Gaussian processes [54, 55, 56, 57, 58), and neural stochastic differential equations (SDEs) [59, 60, 61, 62, 63] have been widely studied for modeling uncertain dynamical systems from data. These models are either deterministic or can capture only

---

single mode of the training dataset's distribution, along with the uncertainty around the mode. On the other hand. Bayesian inference on the parameters of some of these models can. in theory, capture the multimodal distribution of the model parameters. However, classical Monte Carlo-based methods [64, 65, 66, 67] do not scale well with large models and datasets, while variational enference-based methods [68, 69, 70] are limited by the choice of variational family used to approximate multimodal posteriors. To address these limitations, recent works [71, 72, 73, 36] have highlighted the scalability and expressivity of diffusion models when approximating the posterior distribution in Bayesian inference. We leverage such expressivity to capture a multimodal distribution over the parameters of vehicle model expressed as a neural SDE: A model shown in [63] to improve long-tenm prediction accuracy and uncertainty estimate compared to deep Gaussian based models [74. 75].
Our modeling approach is similar to meta-learning methods [76. 77, 78] since the diffusion model learns offline to predict the parameters of a neural SDE model while adapting online to different vehicles and environment conditions. However, in contrast to existing meta-learning approaches where the dataset has task fask-specific labels, we train our model from an unlabeled dataset of vehicle trajectories. In addition, while conditioning the offline-trained diffusion model enables online adaptation, we emphasize that no gradient updates or any sort of regression on the model parameters are performed online for adaptation, as is typically done in meta-learning or online learning [79, 80, 81, 82, 83).
3 Method
We assume access to a dataset T (TI....::T T ([T] of vehicle tra- jectories - = {(1to Mto)..... (xit)11) U [)) where each (#t, we) denotes a state-control pair. We denote by Te:T; = 1 and ,, [u...... Mt4TT the future state and control sequence from time to time + Ty, and by TT,et and Wr, the past sequence from - Tp to t. We also use Te:Ty TTy.t. and Tr,,t:T, to denote the future, past, and full sequence of state-control pairs at time t, respectively. We consider that the dataset's trajectories may be collected from vehicles with different physical and tire specifications, and operating on various road conditions while performing different tasks. Thus, the distribution over its trajectories is complex and challenging model. The dataset is also characterized as unlabelled since no information (per trajectories) about the vehicles and environments is provided for model identification other than the state-control pairs.
Tigaary
W7 (Ty
THI,
Figure 2: Trajectories in T.
We present our approach to learning constreec generative models for autonomous driving at the limits. The main idea is to integrate the structure and the existing physics knowledge of driving at the limits into the design of a neural SDE model parametrized by 8. Then, we train a state-control history conditioned diffusion model, parameterized by tb to output distribution (f ..,,.)) over the neural SDE parameters 6. which is then used to generate the future state trajectories or models to use by MPC controller. However, existing approaches to training diffusion models cannot be directly applied here as they would require unavailable access to a dataset of neural SDE parameters. The proposed approach (Section 3.1) generates such dataset and consistently improves its parameters with respect to the trajectories in T while simultaneously training the diffusion model.
Neural SDEs for modeling. Neural SDEs [84, 85, 59, 60, 63] offer a principled approach for modeling uncertain dynamical systems due to their ability to encode prior physics knowledge from first principles, their calibrated uncertainties, and their expressiveness from using neural networks:
(1) where E X C RR is the state, u E 2 C Rd is the control input, L is the time, W is the
dr 1 u) di + Σ (x, w)dW;
ny-th dimensional Wiener process, and / X x U - Rh= and Σ X' x 2 - R are the parametrized drift and diffusion terms, respectively, and the equation is interpreted an SDE in the Itō sense. Under smoothness assumptions [86, 87] on f "'',, we can efficiently sample a distribution of predicted trajectories via numerical integration of the neural SDE [88, 89]. Assuming approximate Gaussian transitions between discrete times of a numerical SDE integrator, the negative log-likelihonn (NLL) loss of state sequence Te,,, given the controls W:T, can be estimated by
LT,, |I.
5 - + log log(det (ddeeet)))) (2:9)]
Ton TTTT)) == Egg B
(2)
with || = 2T A for 2 E RTx A E RT, and where x := 'd u.) and T6;;; is a sample sequence obtained from the SDE integration for fixed 0 and the initial conditions H, and ucT.
3

---

Algorithm 2 Online MPC Model Sampling
Algorithm Training of the Diffusion Model v
1: Initialize history dataset Thist 0.
1: Initialize D = 0.
2: Initialize best parameters 0' O
2: Compute gloe by solving (3) using SGD,
3: while not terminated do
3: while not converged do
4: Append latest (It, Ut) to Thist
4: Sample TTp:::T, from T.
5: Tgen <- (TTppta IR: Tval + (Teg.Teh) are
5: Sample (Tr,::t,TT,) ) around Trgatiyy
subsets of Thist
6: if Refine then
6: {0r, Jx Po (-((TTTttttt)) using (7).
7:
Update gloc with Po( (o((IN(TT,::)) end if
7: Evaluate scores Junj (8, Trai) for 0 e
8:
{0r, he U {
9:
Optimize for Og using (4) and gloe,
10: Append (TT, 0z) to D.
8: Update Q = {00) j with best scores.
11:
Update & using (5) on batches from D.
9: Send 0ess = argmin (0,Tva)) MPC.
12:
end while
oge
10: end while
3.1
Conditioned diffusion model in parameter space
We provide the main steps of the training process below and summarize them in Algorithm
Initial estimate of the neural SDE parameters. First, we compute the maximum a posteriori estimate of the neural SDE parameters I over the entire dataset T. The training problem is given by
gone f- arggiing Jnnj(0 T), with [m, T) = B(rE [Ja([0, rE:Ty) + Aera)
(3)
where Jau is the negative log- likelihood defined in (2), and T is sampled trajectory from the dataset. The term Atraj controls the regularization term (0) that enforces available prior knowledge on the neural SDE parameters; see Appendix A.4 for details on the design choices in this section.
Parameter dataset generation via local optimization. Next, we use the estimate gloc to iteratively generate dataset 'D = (Trr, 0.)....} } of short state-control trajectories Tr, and model parameters o, that are consistent with (a) the data sequence :r, (b) the immediate future sequence Te,,, and (c) additional sequences : in a neighborhood of the initial data sequence T., The timesteps 4 are sampled from a uniform distribution U of fixed width centered at 6. The local optimal parameter O, that is added to the parameters dataset D is given by O2- argming The ( 0, (loe) with
Thee (Tppt 9 gloo)= == [0, T.. + Es w.(.w) Jan TII-42:1,) )1 + Avoe glocp2 (4)
where the term in Atoe regularizes 0 be close to the estimate @loc, The additional sequences TTyme, help refine the uncertainty estimates of the neural SDE model. Using short sequences Tyste: with 4 in small time window W E R__ around i, helps ensure that they can be explained by a single parameter vector 6r. since properties of the system may otherwise vary over trajectory TTy.:: if TT,,,TT, too long. We optimize for 6g using gradient descent on (4) starting from an initial guess sampled from a Gaussian distribution centered at Oloe. We found that regularizing to the estimate Gloe in the objective (4) stabilizes the optimization process.
Training the diffuuson model. Given the generated neural SDE parameters in D, we update the parameters z of our conditional denoising diffusion model with gradient descent as in [7], on a loss
(v) Ee,,, rrp. (.D.p. J... [lle ( (V7h0, + /1 --ne, k. h ((TT,:))) :)) - -el =]]"], (5)
where Tik == II==1_ (1 10--3p (2) with 9 E 10 being a linear noise schedule to gradually distort the parameters dataset [.. I) is the standard normal distribution, K is the number of diffusion steps, and ev is the denoising neural network predicting the noise e added during the noising process. Here, h is function that maps the history TIpa to a feature space for conditioning the diffusion model:
M(TT,a)) = AATTAAAti Tae WT,et], with AWTØat = [T,+1 TT, .. It - x111),
(6)
and Atrj. Tp: = [(rr,11 - tr,)) (te - 11. We use = 1000 noising steps as in [7].
Iteratively refining the parameter dataset. As the diffusion model training progresses, its predic- tions of the local parameters 0, become more accurate while the initial estimate gloe computed in (3)

---

may become suboptimal initial guess for 8,. Thus, after some number of diffusion training steps, we use the generative model to refine the dataset by sampling a neural SDE parameter that would serve as an initial estimate gloc and regularizer for the local optimization problem (4).
Online diffusion model inference. Algorithm 2 outlines a simple online strategy for sampling the neural SDE parameters conditioned on online measurements, The algorithm maintains the history of state-control pairs and uses it as a source for generating, validating, and scoring the neural SDE parameters. At each time step of the algorithm, a set Tgen of state-action sequences is sampled from the history dataset Thist to generate set of parameters He conditioned on each Tr.st E Tgen- This is done in K steps, by sampling 0x ~ [O I) and refining 0x- ~ N (1p k. ((Tr,.))) TxT) with
1 7 and pa (0, k, V1 3R (0x V1 11-9 7 0 = Bb
Bk. (0x, k =));
- 7
before letting 0, = 0o. Finally, a set Tval S Thist is sampled and used to validate the generated param- eters and compute their scores, defined the loss Tuaj (6, Tvai) in (3) plus 2-norm regularization term that penalizes the distance to the beestt = 5 best previously generated parameters {6P) (8P)asa Whee We found that this last term helps ensure consistent updates during the online inference process.
Ip=1
3.2 Application to autonomous driving at the limits of handling
Physics-constrained neural SDE model. We now introduce the uncertainty-awar and physics- constrained neural SDE model for driving at the limits of handling. We employ the commonly used single-track model [90, 16, 91, 18, 17] as a foundation to describe the nonlinear dynamics of the vehicle. The vehicle position is expressed in curvilinear coordinate system relative to a reference trajectory [14, 19, 12]. Specifically, the position coordinate is described by the distance s along the reference trajectory, the relative heading Ae with respect to reference heading Pret, and the lateral deviation e from the path. The proposed neural SDE model is given by
dr M (z, u) F) (zz u)dt + > (z, u)dW,
(8)
where prior knowledge comes from the matrix (..-)) that depends on vehicle parameters such as the mass m², yaw moment of inertia I rotational inertia of the drivetrain I tire radius R". and distances from the center of gravity to the front and rear axles a and 18 The control input [8, r*] is the steering angle and engine torque, respectively. The state x = [r. V, B, wr, e, AΦ, 6] includes the yaw rate r. velocity V. sideslip angle 3, rear wheelspeed wy, lateral error e, and angular deviation Ań. Lastly, F [F's a "n. F
2r F/0 represents the tire forces between the vehicle and the road. These unknown tire forces F are learned functions of the state and control inputs.
Modeling the dynamic interaction between the tires and the uncertain road surface crucial for accurately controlling vehicle at the limits of handling. To do so, incorporate into neural SDE model a version of the neural-ExpTaane Tanh tire model [15], a physics-ifformed neural tire model that captures the nonlinearities and saturation effects of tire forces, and that has shown to better predict tire forces than previous models used in the literature. We refer to Appendix A.1 for the derivations of (....) and the neural tire force models used in the experiments.
Model predictive control (MPC) for autonomous drifting. The MPC tracks reference trajectory under actuator and actuator rate constraints by solving at each time / the optimization problem
Σn=
" k-1 Qa(3x Brot,t)) Q + So
Qg + + 2e2
minimize ESIHI Gg.
(9a)
subject to Tx+1 SDESolve(Ez, uu; Obeat) Vk = 0, H 2o Iz, ūo. E U, uo. EÚ (9b)
where 2& the state at timestep k. T1:H+1 denotes the state trajectory over the prediction horizon of length H + 1, and uo:H are the corresponding control signals. SDESolve is any differentiable SDE integration scheme, in our case a simple Euler Euler-Maruyan Maruyama method, parameterized using the best parameters Opest. found by Algorithm 2. The states I1.H+1 are thus random variables, and the expectation in (9a) is evaluated using Monte Carlo. We use 2 particles in our experiments. The system is actuated by u4 = uj. where üj is the optimal solution to (9) constrained with Fo Zt.
4 Results
We validate the proposed framework on two vehicles in scenarios with different tires, operating gears, and road conditions. First, verify the capabilities of the conditional diffusion model to capture the
5

---

complexity of the unlabelled dataset and adapt its predictions online (Section 4.1). Second, we show that the model can adapt different tires (Section 4.2). Third, we demonstrate the high performance of the method in diverse scenarios (Section 4.3). Finally, using small amount of data collected on wet surfaces, we show that the framework enables drifting at the limits of handling in heavy rain (Section 4.4). We provide further details on the experiments in Appendix A.
Experimental vehicles. We deploy the approach on Toyota Supra and Lexus LC500, as shown in Figure 1. The Supra is modified with more powerful engine and more responsive actuators, whereas the Lexus is kept with factory settings, making a particularly challenging platform for autonomous drifting. The two vehicles' large differences in dynamics responses make them ideal platforms for evaluating the robustmess and generalization capabilities of our approach. For both vehicles, we use onboard vehicle state estimation using GPS and IMU, and we use the CPU of ruggedized PC to run the diffusion model inference 2Hz and compute control inputs using MPC at 200Hz.
Training dataset. We train the diffusion model on a total of 84 manual and autonomous driving and drifting trajectories from the two vehicles. The duration of each trajectory is between 10 and 90 seconds. It consists of 5 manual driving trajectories pushing the car to the limits of handling, whereas the remaining trajectories are autonomous drifting experiments comprising of failed and successful attempts. The dataset includes driving data in different gears (affecting the effectiveness of the throttle input) and using tires with different physical properties (affecting the vehicle's dynamics and steering input effectiveness). The composition of the dataset is summarized in Figure 3.
Vehicle
Supra (33%)
Lexus LC500 (67%)
Tire
(33%)
(80%)
Gear
(108)
(23%)
(67%)
0%
25%
50%
75%
100%
Figure
Dataset for model training.
Baselines. We compare with neural SDE dynamics models (referred to BaseSDE or Expert depending on context) trained on specific vehicle-tire-ges subsets of the dataset in Figure 3. Each baseline trained with the loss function in (3) and regularization term R(0) encoding prior knowledge about the parameters (m, a, b, Is, Iw: R). The resulting models are Expert since they are optimized for specific scenarios, but they may perform poorly when deployed in different conditions.
4.1 Multimodality and conditioning capabilities of the diffusion model
We condition the model Lexus vehicle dataset (see Figure 3), where the vehi- cle is either accelerating in a straight line (gripping) or drifting (sliding). In Figure 4, we report two pre- dicted parameters (T cf) corresponding to the rear wheel inertia and the maximum total force that can be generated by the rear tires; see Appendix A.1. By conditioning on the straight-line (grip) trajectory, the model returns multimodal parameter distribution due to the lack of information about tire friction prop- erties, as tire forces are not saturated. Interestingly. by conditioning on a trajectory where the vehicle is sliding and tire forces are saturated, the model pre- dicts a tight unimodal distribution of the parameters. The left mode of the two parameters from the gripping phase has collapsed due to sufficient information to infer vehicle properties. This example shows that the proposed conditional diffusion model captures multimodal parameter distributions and adapts its predictions based on the information contained in the trajectory.
trajectories from the
Grip
Stide
1.5
Density
0.5
0.0
log Lu
log max rear force
Figure 4: Parameter distribution predicted by the diffusion model, when conditioned on grip- ping and sliding trajectories.
4.2 Online adaptation to different tires
We further highlight the generalization capa- bility of the proposed method by studying the closed-loop tracking performance of the Lexus when operating with various tires. We report tracking performance on two reference trajec- tories in Table 1. The controller using the Expert model accurately tracks the reference trajectory. In contrast, using the bascline trained only type 3 tire data is insufficient to track the
Table 1: Tracking emro: Lexus with tires type 2.
Donut
Figure-8
RMSE
(m)
(deg)
e (m)
9 (dog)
Expert (Tires 2)
0.35
1.08
0.32
5.17
BaseSDD (Tires 3)
spin
spin
spin
spin
BaseSDE (Tires & 3)
0.51
4.52
1.39
13.63
Dillusion (Tires 2 & 3)
0.31
4.19
0.56
5.38

---

Eass [m))
East (na)
Eias [ta]
Start
12-
-in [m
Nonth
Start
[m/N]
Reference
Diffusion
Expert
0-
[deg]
-33
0.3
0.-
200
200
40
200
240
250
300
400
Distance along path [m]
Distance along path [m]
Distance along path [m]
Figure Drifting the Supra: performance comparison between the Expert and Diffusion models.
reference, and leads to spinning out. This baseline fails initiate the drift, due to differences in the cornering stiffness between the two tires leading to poor prediction of how fast the vehicle will saturate the rear tires. The BaseSDE model trained on data with type and type tires can drift with a higher tracking error. On the donut trajectory, this baseline drifts on circle with larger radius due to a tire force modeling mismatch. Finally, the diffusion model quickly infers the tire properties, which results in adaptive tracking performance that matches Expert performance.
4.3 Drifting performance in different scenarios
We evaluate the framework on a range of scenarios, including reference trajectories that are the dataset, and compare its performance with Expert trained the relevant subset of the dataset.
Drifting results on the Toyota Supra. We report performance in tracking dif- ferent reference trajectories Figure 5 and Table 2. The diffusion model enables drifting maneuvers with tracking perfor-
Table 2: Tracking
the Toyota Supra.
Slalom
(gear 2)
Donot (gear
Figure-8 (gcar
RMSE
e (m))
8 (deg)
(m)
9 (deg)
e (m))
8 (dog)
Expert
0.74
5.03
0.31
3.32
0.32
3.40
Diffusion
0.72
6.32
0.19
2.26
0.57
4.34
mance that is comparable to expert models, while simultaneously having the advantage of being trained on an unstructured dataset. This demonstrates the model's ability to adapt at test time to the specific vehicle setting and road condition based on the online observation. Interestingly. the framework succeeds in accurately tracking the Slalom trajectory (first column Figure 5), which is not part of any maneuvers in the training dataset. Tracking this trajectory requires the vehicle to operate outside of the training data distribution, given the rapid changes between circles of different radii while accelerating during the last transition of the trajectory. We speculate that this ability to generalize results from the prior physics knowledge encoded the neural SDE vehicle model.
Drifting results on the Lexus LC500. Tracking results in Figure 6 and Table show that using the diffusion model en- ables accurate tracking with performance comparable expert models. Again, the
Table 3: Tracking error: Lexus with type tires.
Donut (gear 2)
Donat [gear 1)
Figure-8 (gear 1)
RMSE
(m)
8 (deg)
(m)
3 (deg)
e (m)
8 (dog)
Expert
0.48
3.44
0.38
6.39
0.32
3.27
Dilfusion
0.82
4.70
0.29
4.48
0.34
3.79
proposed method is capable of performing donut trajectory in second gear, although no second-gear trajectory from the Lexus is in the dataset. Moreover, only unsuccessful Figure-8 trajectories on the Lexus are in the dataset, yet both methods using the expert and diffusion models succeed in tracking this trajectory, thanks to the physics structure encoded in the neural SDE model.
4.4 Drifting in low-friction conditions using limited data
7

---

East [m)
Elss
East
10 11
North [m]
Start
Start
North [m]
V Lmns
Reference
Diffosion
Expern
Distance along
Distance along pach [m]
Distance along
Figure 6: Drifting the Lexus: performance comparison between the Expert and Diffusion models.
Lastly, we augment the dataset with 3 manual drifting trajectories collected in the rain and 4 au- tonomous drifting donut trajectories on a wet surface with failed and successful attempts on the Lexus diffusion model and report the performance of the resulting controller
vehicle. Then, we retrain deployed in heavy rain in Figure 7. The proposed approach is capable of drifting with only 1.47 m as the lateral RMS error (e) and 4. 79 deg as the RMS slip angle (3) error.
Drifting in such rainy conditions is particularly chal- lenging. Indeed, accurate friction modeling critical to successfully initiating the drift without spinning out, especially using a commercial vehicle such the Lexus. Stabilizing the vehicle in rainy conditions particularly difficult due the increased effect of small friction variations on the handling characteris- ties of the vehicle, and because friction parameters vary over space due to the terrain drying unevenly over time. These results indicate that single gener- ative model, trained on a majority of data collected on nigh-friction surfaces, has the potential to enable reliable autonomous driving at the limits in both high and low-friction conditions.
V [mub]]
18 [deg]]
along path [m]
Figure Tracking performance in heavy rain
a donut trajectory when drifting the Lexus.
Conclusions
We propose thysics-informed generative vehicle model for autonomous driving at the limits of handling. By decoupling model inference and control, this hierarchical approach combines the expressiveness diffusion model with the high-rate replanning and reliability of model predictive control. Through extensive autonomous drifting experiments on a Toyota Supra and I Lexus LC500, we demonstrate that a single conditional diffusion model, trained on unlabelled trajectories from both vehicles operating in various conditions, enable adaptive, robust, and real-time autonomous driving at the limits of handling.
Limitations and future work. Although the diffusion model predicts multimodal distribution of parameters of the neural SDE vehicle dynamics model, the current model predictive controller only uses one predicted parameter set for control at time. Fully reasoning over predicted distribution using risk-sensitive algorithms would potentially lead to additional robustness and inform data collection via active exploration to best reduce uncertainties online. Finally, while only validated on drifting tasks, the generality of the proposed hierarchical method and the experiments indicate that the approach could potentially be used in other autonomous driving and robotics applications.

---

Acknowledgments
We would like to thank the platform research team at Toyota Research Institute for their support with the test platforms and experiments. Special acknowledgment to Phung Nguyen and Steven Goldine for facilitating the experiments and making possible validate our framework under various conditions, and Jenna Lee for facilitating data logging and processing. We would also like to thank Yusei Sakamoto at Toyota Motor Corporation for the tremendous assistance setting up our framework on their test platform, and creating an environment to validate our approach on wet surfaces and rainy conditions.
References
[1] A. Gray, Y. Gao, T. Lin, J. K. Hedrick, H. E. Tseng, and F. Borrelli. Predictive control for agile semi- -utt-oouonomuus ground vehicles using motion primitives. In 2012 American Control Conference (ACC), pages 4239-4244, 2012.
[2] T. Zhao, E. Yurtsever, and G. Rizzoni. Justifying emergency drift control for automated vehicles. Pape--appr,, (5(24):141-144) 2022.
[3] T. Gordon and M. Lidberg. Automated driving and autonomous functions on road vehicles. Vehicle System Dynamics, 53(():958-99)) 2015.
[4] Y. Goh, M. Thompson, J. Dallas, and A. Balachandran. Nonlinear model predictive control for highly transient autonomous drifting. 15th International Symposium Advanced Vehicle Control, 2022.
[5] T. P. Weber and J. C. Gerdes, Modeling and control for dynamic drifting trajectories. IEEE Transactions on Intelligent Vehicles, 2023.
[6] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2255,2265.22 2015.
[7] J. Ho, A. Jain, and P. Abbeel. information processing systems, 33:6840 6851, 2020.
Denoising diffusion probabilistic models. Advances in neural
[8] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[9] Y. Song, Sohl-Dickstein, generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
Kingma, Kumar, S. Ermon, and B. Poole. Score-based
[10] G. Williams, Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou. Aggressive driving with model predictive path integral control. In 2016 IEEE International Conference Robotics and Automation (ICRA), pages 1433-1440, 2016.
[11] J. Dallas, M. Thompson, J. Goh, and A. Balachandran. A hierarchical adaptive nonlinear model predictive control approach for maximizing tire force usage in autonomous vehicles. Field Robotics, 3(1):222-244) 2023.
[12] K. Subosits and J. C. Gerdes. Impacts of model fidelity on trajectory optimization for autonomous vehicles in extreme maneuvers. IEEE Transactions on Intelligent Vehicles, 6: 546-558, 2021.
[13] J. Dallas, M. P. Cole, P. Jayakumar, and T. Ersal. Terrain adaptive trajectory planning and tracking deformable terrains. IEEE Transactions on Vehicular Technology, 70(1 1255- 1 11268, 2021.
[14] J. Y. Goh, T. Goel, and J. C. Gerdes. Toward automated vehicle control beyond the sta- bility limits: Drifting along a general path. Journal of Dynamic Systems Measurement and of The Asme, 142, 2019.

---

[15] F. Djeumon,, J. Y. Goh, U. Topcu, and A. Balachandran. Autonomous drifting with 3 minutes of data via learned tire models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages -70, 2023.
[16] R. Rajamani. Vehicle dynamics and control. Springer Science & Business Media, 2011.
[17] P. Polack, F. Altché, B. d' A'Anrééa Novel, and A. de La Fortelle. The kinematic bicycle model:
A consistent model for planning feasible trajectories for autonomous vehicles? 2017 IEEE Intelligent Vehicles Symposium (IV), pages 812-818, 2017.
[18] J. Kong, M. Pfeiffer, G. Schildbach, and F. Borrelli. Kinematic and dynamic vehicle models for autonomous driving control design. 2015 IEEE Intelligent Vehicles Symposium (IV), pages 1094-1099, 2015.
[19] Y. Goh and J. Gerdes. Simultaneous stabilization and tracking of basic automobile drifting trajectories. 2016 IEEE Intelligent Vehicles Symposium (IV), pages 597-602, 2016.
[20] N. A. Spielberg, M. Brown, and J. C. Gerdes. Neural network model predictive motion control applied to automated driving with unknown friction. IEEE Transactions Control Systems Technology, 3065):1934-1455 2021.
[21] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and Sutskever. Zero-shot rext-to-image generation. In International conference on machine learning, pages 0821-8831, 2021.
[22] C. Saharia, W. Chan, S. Saxena, L. Li, Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35: 36479-36494, 2022.
[23] A. Vahdat, F. Williams, Z. Gojeic, O. Litany, S. Fidler, K. Kreis, et al. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems, 35:1 35:10021-10039, 2022.
[24] L. Zhou, Y. Du, and J. Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5826-5835, 2021.
[25] C. Chi, S. Feng Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[26] A. Ajay, Y. Du, A. Gupta, Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional gen- erative modeling all you need for decision-making?? International Conference on Learning Representations, 11, 2023.
[27] B. Kang, X. Ma, C. Du, T. Pang, and S. Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[28] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, Hofmann, et Imitating human behaviour with diffusion models, arXiv preprint
10677, 2023.
[29] T. Gu, G. Chen, J. Li, C. Lin, Y. Rao, J. Zhou, and J. Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17113-17122, 2022.
[30] C. Jiang, A. Cormman, C. Park, B. Sapp, Y. Zhou, D. Anguelov, et al. Motiondiffuser. Con- trollable multi-agent motion prediction using diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9644-9653, 2023.
[31] D. Rempe, Z. Luo, X. Bin Peng, Y. Yuan, K. Kitani, K. Kreis, S. Fidler, and O. Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13756-13766. 2023.
10

---

[32] J. M. L. Alcaraz and N. Strodthoff. structured state space models. arXiv preprint arXiv:2208.09399 2022.
time series imputation and forecasting with
[33] Y. Tashiro, J. Song, Y. Song, and S. Ermon. Csdi: Conditional score-based diffusion models for probabilisticc time series imputation. Advances in Neural Information Processing Systems, 34: 21804, 2021.
[34] B. Holzschuh, S. Vegetti, and N. Thuerey, Solving inverse physics problems with score matching. Advances in Neural Information Processing Systems, 36, 2023.
[35] G. Kohl, L.-W. Chen, and N. Thuerey. Turbulent flow simulation using autoregressive condi- tional diffusion models. arXiv preprint arXiv:2309.0174 2023.
[36] Q. Liu and N. Thuerey. Uncertainty-awant surrogate models for airfoil flow simulations with denoising diffusion probabilistic models. American Institute of Aeronautics and Astronautics, pages 1-22, 2024.
[37] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 10684-10695, 2022.
[38] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural ordinary differential equa- tions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6572-6583, 2018.
[39] C. RRackauckas, Y. Ma, J. Martensen, C. Warner, K. Zubov, R. Supekar, D. Skinner, A. Ramadhan, and A. Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001 004385, 2020.
[40] F. Djeumou, C. Neary, E. Goubault, S. Putot, and U. Topeu. Neural networks with physics- informed architectures and constraints for dynamical systems modeling. In Proceedings of The 4th Annual Learning for Dynamics and Control Conference, volume 168, 2022.
[41] M. Cranmer, S. Greydanus, S. Hoyer, Battaglia, D. Spergel, and S. Ho. Lagrangian neu- ral networks. In International Conference on Learning Representations, 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
[42] S. Greydanus, M. Dzamba, and Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, 2019.
[43] B. O. Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences, 17(5)311-311,, 1931.
[44] I. Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 41:309-325, 2005.
[45] I. Mezić. Analysis of fluid flows via spectral properties of the koopman operator. Annual review of fluid mechanics, 45:357-378, 2013.
[46] C. W. Rowley, I. Mezić, S. Bagheri, P. Schlatter, and D. S. Henningson. Spectral analysis of nonlinear flows. Journal of fluid mechanics, 641:115-127, 2009.
[47] J. L. Proctor, S. L. Brunton, and J. N. Kutz. Generalizing koopman theory to allow for inputs and control. SIAM Journal on Applied Dynamical Systems. 17(1):909_9 930, 2018.
[48] S. L. Brunton, M. Budišić, E. Kaiser, and J. N. Kutz. Modern koopman theory for dynamical systems. arXiv preprint arXiv:2102. 11022.22200 12086, 2021.
[49] J. Proctor, S. L. Brunton, and N. Kutz. Dynamic mode decomposition with control. SIAM Journal on Applied Dynamical Systems. 151))1142-16,, 2016.
[50] E. Kaiser, J. N. Kutz, and S. L. Brunton. Sparse identification of nonlinear dynamics for model predictive control the low-data limit. Proceedings of the Royal Society A, 474(2219): 201 80130335, 2018.
11

---

[51] M. Korda and I. Mezić. Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control. Automatica, 93:149-160, 2018.
[52] Coulson, Lygeros, and Dörfler. Data-enabled predictive control: In the shallows of the deepc. In 2019 18th European Control Conference (ECC), pages 307-312, 2019.
[53] H. J. van Waarde, M. K. Camlibel, and M. Mesbahi. From noisy data to feedback controllers: Nonconservative design via a matrix s-lemma. IEEE Transactions on Automatic Control, 67 (1):162-175, 2020.
[54] C. E. Rasmussen. Gaussian processes in machine learning. Springer, 2003.
[55] C. K. Williams and C. E. Rasmussen. Gaussian processes for machine learning. MIT press Cambridge, 2006.
[56] L. Song, J. Huang, A. Smola, and K. Fukumizu. Hilbert space embeddings of conditional distributions with applications dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 961-968, 2009.
[57] J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, M. Prabhat, and R. Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pages 2171-21111 2015.
[58] A. J. Thorpe, C. Neary, F. Djeumou, M. M. Oishi, and U. Topeu. Physics-informed kernel embeddings: Integrating prior system knowledge with data-driven control. arXiv preprint Stiv:2301.035653 2023.
[59] B. Tzen and M. Raginsky. Neural stochastic differential equations: Deep latent gaussian models the diffusion limit. arXiv preprint S1Kiv:19055098833 2019.
[60] Kidger. On neural differential equations. PhD thesis, University of Oxford, 2021.
[61] X. Li, T.-K. Wong, R. T. Chen, and D. K. Duvenaud. Scalable gradients and variational inference for stochastic differential equations. In Symposium on Advances in Approximate Bayesian Inference, 2020.
[62] Lew, S. Singh, M. Prats, Bingham, J. Weisz, B. Holson, X. Zhang, V. Sindhwani, Y. Lu, F. Xia, et al. Robotic table wiping via reinforcement learning and whole-body trajectory optimization. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7184-7190, 2023.
[63] F. Djeumou, C. Neary, and U. Topeu. How to learn and generalize from three minutes of data: Poystic-c and uncertainty-awan neural stochastic differential equations. In Conference Robot Learning, pages 577-601, 2023.
[64] W. K. Hastings. Monte carlo sampling methods using markov chains and their applications. Biometrika, 57:97-109, 1970.
[65] S. Chib and E. Greenberg. Understanding the metropolis-hastings algorithm. The american statistician, 49(4):327-335, 1995.
[66]
R. M. Neal al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(1 1):2, 2011.
[67] M. D. Hoffmman, A. Gelman, et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. Journal of Machine Learning Research, 15(1):1593-1233 2014.
[68] M. D. Hoffman, M. Blei, C. Wang, and Paisley. Stochastic variational inference. Journal of Machine Learning Research, 2013.
[69] A. Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 201
[70] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 1 112(518):859-87, 2017.
12

---

[71] L. Baldassari, A. Siahkoohi, J. Garnier, K. Solna, and M. V. de Hoop. Conditional ccore-based diffusion models for bayesian inference in infinite dimensions. Advances Neural Information Processing Systems, 36, 2024.
[72] B. Kawar, G. Vaksman, and M. Elad. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757-21769) 2021.
[73] Y. Song, L. Shen, L. Xing, and S. Ermon. Solving inverse problems prolemss in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005, 2021.
[74] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[75] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
[76] J. Schmidhuber. Evolutionary principles in self-referential learning, learning how to learn: the meta-meta-... hook. PhD) thesis, Technische Universität München, 1987.
[77] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-lcarning with memory-augmented neural networks. In International conference on machine learning, pages 1842-1850, 2016.
[78] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, 2017.
[79] L. Bottou. Online learning and stochastic approximations. Online learning in neural networks, 17(9):142, 1998.
[80] M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109-165, 1989.
[81] Kivinen, A. Smola, and R. C. Williamson. Online learning with kernels. IEEE Transactions on Signal Processing, 52:2165 2176, 2001.
[82] T. Lew, A. Sharma, J. Harrison, A. Bylard, and M. Pavone. Safe active dynamics learning and control: A sequential exploration exploitation framework. IEEE Transactions on Robotics, 38: 2888-2907, 2020.
[83] M. P. Deisenroth, D. Fox, and C. Rasmussen. Gaussian processes for data-cfficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37: 408 423, 2015.
[84] R. L. Stratonovich. A new representation for stochastic integrals and equations. Siam Journal on Control, 4:362-371, 1966.
[85] K. Itô. On stochastic differential equations. American Mathematical Society, 1951.
[86] B. Øksendal. Stochastic differential equations. Springer Berlin Heidelberg, 2003.
[87] H. Kunita. Stochastic flows and stochastic differential equations. Cambridge university press,
1997.
[88] G. N. Milstein. Numerical integration of stochastic differential equations. Springer Science Business Media, 1994.
[89] P. E. Kloeden, E. Platen, and H. Schurz. Numerical solution of SDE through computer experiments. Springer Science & Business Media, 2002.
[90] B. Paden, M. Cáp. S. Z. Yong, D. S. Yershov, and E. Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on Intelligent Vehicles, 1:33-55, 2016.
13

---

[91] P. Falcone, F. Borrelli, J. Asgari, H. E. Tseng, and D. Hrovat. Predictive active steering control for autonomous vehicle systems. IEEE Transactions on Control Systems Technology. 15:566-580, 2007.
[92] J. Bradbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, Wanderman-Miin Minne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
com/ google/jax.
[93] D. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014.
[94] Q. Li, Y. Zhou, Y. Liang, and P. K. Varshney. Convergence analysis of proximal gradient with momentum for nonconvex optimization. In International Conference Machine Learning, pages 2111-2119, 2017.
14

---

A Appendix
This section provides additional details on the neural SDE vehicle model, the experimental vehicles and training dataset, the expert models used as baselines, the diffusion model training, and the model predictive control formulation.
We implement all the numerical experiments (training the models and the gradient-based model predictive control solver) using the Python library JAX [92] to take advantage of its automatic differentiation and just-in-time compilation features. We use Python 3.8.5 for the experiments and train all our models on a laptop computer with an Intel()) (R) W-11855M CPU (base frequency 3.30GHz), 12 cores, 32 GB of RAM, and GeForce RTX 2060, TU10.
A.1 Physics-inspired neural SDE vehicle model
We employ the commonly used single-track model [90, 16, 91, 18, 17] as foundation to describe the nonlinear dynamics of the vehicle. The vehicle position is expressed in a curvilinear coordinate sys- tem relative reference trajectory [14, 19, 12], shown in Figure 8. Specifically, the position coordi- nate described by the distance along the path, the relative heading Ań with respect to planned course deef, and the lateral deviation from the path. For simplicity, we assume only the steering and throttle are used for autonomous drifting and do not include brakes in the control inputs and dynamics. The pro- posed neural SDE model is given by
World East
PATH TANGENT
Figure 8: Single-track model of a vehicle on reference path.
12 cos(6) to re, sim(f) re
16
-F sin(8 F ons - (1) po, sin(3)) F (ns(
V
m
+ Σ (2,
(10)
Fgg castb- + sin(d +)+ pe ons B Fb, s
m²V
GE (T*)
Wy
18
where vehicle-specific parameters such as the mass m.. yaw moment of inertia 1 rotational inertia of the drivetrain 19- the tire radius R and the distances from the center of gravity to the front and rear axles d and 1 are included in the neural SDE parameters 0 to learn. The control input [8, r is the steering angle and engine torque, respectively. E*(r)) is parameterized polynomial function that maps the engine torque to the wheel torque through the gear ratio G. The state x = [r V, _3, Wr: e, AΦ, includes the yaw rate r, velocity V. sideslip angle 3, rear wheelspeed wr. lateral error e, and angular deviation Aø. The evolution of the path-dependent variables e, Ad, and well described by basic kinematics
de = V "sinnA
(11)
V cos( AØ) dt.
ds 1- --eriee (s)
(12)
d(AØ) = (B +r - Kire (8)8)d(,
(13)
where (s) is the curvature of the reference path, and we use and as an abuse of notation for the drift terms of 3 and s, respectively. Lastly, the unknown lateral and longitudinal tire forces F uff
F8 gr. and F are parametrized and learned as functions of the state and control inputs.
15

---

We propose to incorporate into our neural SDE model a version of the recently proposed neural- ExpTann [15] tire mode parameterized by
Fg/ "apTTanh'
8 (tan(ay); featz), Fist = ExpTann
(tan?(a,) (aannaaanntta((aaan + cjo; featz),
NN (ap, Fr)
Fd 0, R' cos 3
yr P2r
INN or)l
Ccos 3
(14)
V sin 3 + arr
V Vsn 3 - b"r
as atan
6, 0r atan
V cos 3
V cos
where ExpTanh (z; feat) = el + Glel tanh (2(=-63) is such that (c))2_1 NN (feat) is the output of neural network with input feat and satisfying cj of 2 0 (enforced via exponential function on the last two outputs of the neural network). Besides, o, is the slip ratio, o and a, are the slip angles for the front and rear tires, feat, [r, V. and feats- (r, V, w, are features for the two neural-ExpTaanl models, and of and NN are learned to approximate the coupled effect between the longitudinal and lateral tire dynamics when the vehicle is sliding and accelerating at the same time. We note that our model has four neural networks: NN6 for the coupled effect, NN; and NN for front and rear neural-Exxe Exp'aaan tire models, and n for the diffusion term. Lastly, we emphasize that although the proposed neural model assumes rear-wheel drive vehicles, it can be straightforwardly extended to other drive configurations.
Modeling details. We use diagonal matrix to represent the noise scale Σ This design choice greatly reduces the computation of the losses (2), (3), and (4) at the cost of possibly limiting the expressivity of the model by neglecting correlations between states. However, the experiments show that the model can still capture the complex dynamics of the vehicle and enable reliable performance on diverse drifting maneuvers. The vehicle parameters m 8 1 1 R2 a", and 6d are learnable scalar values optimized during training. We refer to Appendix A.3 for prior knowledge enforced on these vehicle parameters during the training of expert models, but not during the diffusion model training. The engine torque function E (r") is linear function with learnable parameters. The parameter of is learnable scalar value, and the neural networks NN: NNŞ, and NNŞ are feedforward neural networks with two hidden layers of 6 neurons each and tanh activation functions. We do not perform any preprocessing on the training dataset and learn from raw and noisy vehicle trajectories.
A.2 Experimental vehicles and training dataset
Figure 9: Experimental vehicles used in the study. Left: Toyota Supra. Right: Lexus LC500.
Experimental vehicles. We deployed our framework on a Toyota Supra and a Lexus LC500, illustrated in Figure 9. The Supra is prototype vehicle that has been heavily modified to become an autonomous drifting platform, while the Lexus a commercial 2019 LC500 with its powertrain, drivetrain, and suspension unmodified from the dealership. The Supra has been modified with a 3L inline-six engine capable of outputting 380hp. The engine has been outfitted with an upgraded turbocharger that can provide an additional 300hp, for a total of 680hp. The steering system is outfitted with both hydraulic assist and electric power-assisted steering, providing high-performance wiee abilities with up 56.7Nm of torque. The vehicle has also been modified to provide
16

---

trake-by-wirg and strottle-by-win capabilities. These modifications make the platform extremely suitable for (autonomous) Formula drift, as they allow for precise and fast control of the vehicle's dynamics. The Lexus, on the other hand, has lower- uwwerpperformance actuators and is not designed to be used as drifting vehicle. Ultimately, these two platforms have completely different dynamies, which makes them ideal for evaluating the robustness and generalization capabilities of our approach.
We use an Oxford Technical Systems (OxTS) RT4003 v2 KTK-GPS/IMLL system for localization and vehicle state estimation for both vehicles. The MPC controller is implemented on an Intel Xeon E-2278GE (base frequency 3.30GHz) CPU Linux computer mounted on board the vehicles. The computer communicates with low-level PID controller implemented on a dSpace MicroAutoBox.II (DS1401) track desired steering angle å and engine torque T The MicroAutoBoxII receives commands from the MPC controller via UDP and sends actuator commands to the original equipment manufacturer's 's steering and engine electronic control units. All data are synchronized and recorded at frequency of 62.5Hz on the Lexus and 100Hz on the Supra.
Training dataset. We build a dataset of manual and autonomous driving and drifting trajectories on a closed circuit from both vehicles. The dataset contains a total of 84 trajectories, each trajectory with duration between 10 and 90 seconds. has 5 trajectories collected from manual driving with the intent of pushing the car to the limits of handling without any specific path-tracking maneuvers planned. The remaining trajectories are from autonomous drifting experiments with from the Supra and the rest from the Lexus. The supra dataset contains failed and successful attempts at performing donut maneuvers in first gear and Figure-8 maneuvers in second gear. The Lexus dataset contains attempts at performing donut maneuvers in first gear and very limited (all failed) attempts at performing "Figure-8" maneuvers in first gear. No second-ges gear drifting trajectories were provided in the training dataset for the Lexus. Additionally, 7% of the trajectories from the Lexus were collected with Tire 2. as opposed to Tire used in the rest of the dataset. A notable difference between the two sets of tires is their cornering stiffness, which makes drift initiation strategies and tire dynamics different between the two sets of tires.
Training dataset for drifting on heavy rain. We augmented the above training dataset with 7 additional drifting trajectories collected on a wet track with a vehicle identical to the Lexus LC500. 3 of the trajectories were manually collected, while the remaining 4 were collected autonomously. The autonomous trajectories were collected using second-ges drifting maneuvers only on a donut trajectory. Despite the limited number of trajectories and restriction to second-gear drifting, show in Section 4.4 that the diffusion model trained on this dataset generalizes to drifting in heavy rain on first-gear donut trajectory.
A.3 Expert models training
Table 4: Prior parameters for the expert models.
Vehicle
m (kg)
Lg (kg-m")
Iu (kg-m²)
R (m)
a (m)
b (m)
Toyota Supra
2048
3675
0.368
1.345
1.522
Lexus LC500
1476
2241
6
0.323
1.239
1.209
We recall that the expert models in the experiments are neural SDE models trained on specific vehicle-tire subsets of the dataset. Given a subset Texp of the training dataset T, we train the expert models by minimizing the negative log- ligeliillhoood Junj (0, Texp) defined in (3) with respect to the neural SDE parameters 0 defined in Appendix A. The regularization term R()) encodes a Gaussian prior on the vehicle parameters m². 1 Ihe Rd a², and ba. For each parameter, we use a Gaussian prior with mean equal to the known (estimated) parameter value for the specific vehicle and standard deviation of 1. The resulting R() is given by
R(0) =(m² - m)² + (1- - 1,)² + ( - 12) (R RR) -aa (a'
(6" 6)2,
(15)
where the parameters values m. Ig. Iw. R, and b for each vehicle are provided in Table 4. The expert models are trained using Adam optimizer [93] with a learning rate of 10-3 and a batch size
of 64. We use Atraj 10-2 for the regularization term in the loss function (3). During training, we discretize the sum in the expression of Jnl (see (2) for the expression and (3) for the loss function) into
sum of Nr = 20 discrete time steps to..... ... tN. where to = t, T₁ tN to. and 4-11 At,
17

---

for i = 1...... N₁. We set At, = (1,))) where At typically 0.01 for trajectories on the Supra and 0.016 for trajectories on the Lexus. We randomize the time steps At, during training to improve the model's generalization when evaluated with an integration scheme that uses stepsizes other than the training stepsizes. This is typically the case when using the model in model predictive controller; see Appendix A.5. We note that the model learns to predict state-action sequences with varying lengths Tr between 0.4 seconds and 1.96 seconds. Lastly, we use 5 particles of the neural SDE to compute the expectation in the expression of Jall.
A.4 Diffusion model training and online sampling
Initial estimate of the neural SDE parameters. We train the initial estimate gloe of the neural SDE parameters in similar manner to the expert models in Appendix A.3. The main difference is that we use the full training dataset T instead of the vehicle-specific subset Texp: and thus we do not enforce any prior knowledge on the vehicle parameters as in (15). Specifically, the regularization term (0) is now enforcing an uninformative Gaussian prior with mean and standard deviation 1 on all the neural SDE parameters 0 as follows
Σ No
R(0)
i- (0.)²,
(16)
where No is the number of parameters in 0, and @ in defined as in Appendix A.3.
Parameter dataset generation via local optimization. The hyperparameters that define this step of the diffusion model training are the regularization parameter Aloc. the time window W, the history length Tp. the future trajectory length Tr, and the number sequences :Tu used to compute the expectation in (4). We set Ato = 10 -,033000 w = 10 seconds, and the number of sequences :tt: to 5. In similar manner to how we chose the time steps in the training of the expert models (see Appendix A.3), we discretize a and :Tp into N, = 10 discrete time steps with the same randomization of the step size At; as in Appendix A.3. Thus, the maximum length of the history sequence is Tр = 0.96 seconds. Besides, we keep the future trajectory length Tę to be the same as in the training of the expert models, i.e., with N₂- 20 discrete time steps and maximum length of 1.96 seconds. We use 5 particles of the neural SDE to compute the expectation in the expression of Jan-
We optimize the neural SDE parameters 0 of the loss function (4) using gradient descent with Nesterov acceleration and an adaptive learning rate through Armijo line search. We set the maximum number of iterations to 1000 and the initial guess for the learning rate and neural SDE parameters to be respectively 0.01 and gloc.
Diffusion model training, We follow the procedure described in [7] to train all our diffusion models. The model 6, defining the generative process is represented as a standard feedforward neural network with three hidden layers of 256 neurons each. We use sinusoidal positional encoding of the diffusion step h (see (5)) as an input to the neural network defining Ey. instead of using directly the imput. The encoding is done by scaling the diffusion step and concatenating its sine and cosine to the input of a feedforward neural network with two hidden layers of 32 and 16 neurons each and swish as the activation function. We use K = 1000 denoising steps, and linear noise schedule 33 E (0. 1). where 33 = 0.0001 + 0.02i/ K for = O..... K. We use Adam optimizer with learning rate of 10-4 and batch size of 32 train the diffusion model. Additionally, perform 50 gradient updates of e,) for each step of Algorithm 1.
Iteratively refining the parameter dataset. Given the history sequence Tr, we use the diffusion generation process defined in (7) to obtain set of parameters {0p }ب=ر conditioned on .r Then, we use the future sequence Tr to select the best parameter in terms of the negative log-likelihood OO, TT,,):T,) ) with 5 particles of the neural SDE to compute the expectation in its expression. The obtained best parameter is then used update gloc for better initialization and regularization of the local optimization problem (4), In our experiments, we refine gloc at every step of Algorithm I only after the initial 20000 steps of the main training loop.
Online diffusion model inference. In Algorithm 2, we use a sliding window to deal with the growing size of the dataset Taist and to account for changing vehicle-road properties or environment conditions. The maximum size of the sliding window Thist is typically set to 30 seconds worth of
18

---

driving data. During online sampling for model predictive control, we randomly sample the set Igen to contain history sequences TTt and the set Tval - Thist. contain 30 sequences TT:T, of the current history dataset Thist. Specifically, by indexing each state-action pair , Mt, with the corresponding discrete time, we can define a discrete distribution to sample sequences + by sampling time indexes i and using the corresponding state-action as the endpoint of the sequence of length Tp. In our experiments, we use an exponential distribution with a higher mass on the latest time indexes in Thist to generate Tgen. Then, to generate Tral. we pick the future sequences Tr,T, corresponding to the sequences of Tgen and sample the remaining validation sequences Tt:T; in the same manner Tgen. The sequences in the validation dataset are selected to be as close as possible to the latest time in the history dataset. We use the diffusion model to generate a total of 100 parameters {8t he conditioned on the sequences in T genn+ and select the best parameter according to Algorithm 2.
A.5
Model predictive control formulation
We use a custom proximal gradient-based solver with Nesterov accelcration and Armijo line search, inspired by the approach in [94], optimize the MPC problem 9a-9b. The state constraints, if any, and control rate constraints are enforced using slack variables, and the proximal operator for projecting the slack variables onto the feasible set. We use first-order approximation to compute the control rate as in شة (ux+1 /x Atp. On the other hand, the box constraints on the control input are simply enforced by projection onto the set at each iteration of the proximal gradient- based solver.
Reference trajectories. The reference trajectories for the maps are generated offline via the quasi- equilibrium strategy proposed in [19]. We start with few waypoints in terms of the desired curvature Fref and sideslip angle Bret as a function of path distance s. Then, for each point on the path, an equilibrium point is computed using the single-track bicycle model and the conditions к Kref- = Bret, o Keet V. and V = f 0, yielding the fine- fine-graine grained reference vehicle state Tref- We emphasize that the model used to generate the reference trajectories differs significantly from the neural SDE model used in the MPC controller. We reduce the over-reliance possibly infeasible reference trajectory by using a cost function that penalizes the deviation from only in the sideslip angle 3 and wheel speed wy, see (9a)-(9b).
Lexus LC500. We use Q₁ 120, Q. 2.0, Qa 60.0, Q3 5, and Q+ 10-6, The control set given by U [- 0.52, 0.52] [-1, 400] while the control rate set is given by it [-0.9, 0.9) x (-3000, 400). The problem is optimized over a horizon H = 30 with 25 short time steps of 0.05s and 5 long time steps of 0.15s. Thus, the total lookahead horizon amounts to 2s.
Toyota Supra. We use Q3 = 70, Q. = 3.0, Q = 30.0, Q; = 1, and Q+ = 10-7. The control set is given by U 0.75, 0.75] [-50, 350 while the control rate set is given by ú = 2 2] x -000, 2000]. The problem is optimized over horizon H = 30 with 25 short time steps of 0.05s and 5 long time steps of 0.15s. Thus, the total lookahead horizon amounts 2s.
A.6 Details on the drifting experiments
In this section, we provide additional details on the drifting experiments conducted with the Toyota Supra and Lexus LC500 vehicles. We show the full vehicle state and control evolution when drifting on the various trajectories and road conditions, and when equipped with different tires:
Figure 10 shows the Lexus LC500 drifting on first-gear donut trajectory with Tire 3.
Figure 11 shows the Lexus LC500 drifting on first-gear Figure-8 8 trajectory with Tire 3.
Figure 12 shows the Lexus LC500 drifting on second-gear donut trajectory with Tire 3.
Figure 13 shows the Toyota Supra drifting on a first-gear donut trajectory.
Figure 14 shows the Toyota Supra drifting on a second -garr Figure-8 trajectory.
Figure 15 shows the Toyota Supra drifting on a second-gear slalom-like trajectory.
Figure 16 shows the Lexus LC500 drifting on first-gear donut trajectory with Tire 2.
Figure 17 shows the Lexus LC500 drifting on first-gear Figure-8 trajectory with Tire 2.
Figure 18 shows the Lexus LC500 drifting on first-gear donut trajectory on heavy rain.
19

---

Reference
Expert
Diffusion
V [mms]]
[rad/s]
(deg]
[dleg]
A6 [rad]
0.1
e [m]
-0.1
R [mis]
300
[Nm]
1001
Distance along path [m]
Distance along path [m]
Figure 10: Lexus drifting on first-gear donut trajectory with Tire 3.
20

---

Reference
Expert
Diffusion
v [mmi)
Trad//)
(deg)
[degl]
0
-5
-20
0.1
Ae [rad]
e [m]
-0.1
300
R [mis]
[Nm]
1011
10
0-
150
200
250
300
150
200
250
300
Distance along path [m]
Distance along path [m]
Figure 11: Lexus drifting on first-gear Figure-8 trajectory with Tire 3.
21

---

Reference Expert
Diffusion
V [mis]
[rad/s]
0.5
10
0.0
B [deg]
[deg]]
-20
0.1
M
AΦ [rad]
[m)
0.1
400
R ob, [m/s]
r [Nm]
200
0-
100
200
500
400
100
300
400
Distance along path [m]
Distance along path [m]
Figure 12: Lexus drifting on second-gear donut trajectory with Tire 3.
22

---

Reference Expert
Diffusion
[mmss]
[radis]
1.0-
0.5
0.0
0-
B3 [deg]
[deg]
-00
40
1.0
m
[m)]
0.5
de [rad]
0.1
20
200
R [m/s]
r' [Nm]
10
200
300
350
200
250
300
Distance along path [m]
Distance along path [m]
Figure 13: Toyota Supra drifting on first-gear donut trajectory.
23

---

Reference Expert
Diffusion
V [mis]
16
[rad/s]
25
B [deg]
8 [deg]
0-
-25
M
A@ [rad]
[m)]
-0.1
R Oh, [m/s]
[Nm]
0-
200
200
300
400
500
200
400
600
Distance along path [m]
Distance along path [m]
Figure 14: Toyota Supra drifting second-geat Figure-8 trajectory.
24

---

3
3
Reference
Expert
Diffusion
16
V [mis]
[rad/s]
12-
25
25
B [deg]
8 [dleg]
0-
225
-25
1
AΦ [rad]
[1)
-0.1
R Ob, [m/ss]
20-
[Nm]
200
10-
200
150
200
230
300
350
400
450
150
200
250
300
350
400
450
Distance along path [m]
Distance along path [m]
Figure 15: Toyota Supra drifting second-geat gear slalom-like trajectory.
25

---

Reference Expert (Tires
Diffusion (Tires 3)
Bas Tires 3)
[mvbs]
[[nd/t]]
20w
[deg]]
8 (degg]
20
-40
-00
0.5
0.1
[mm]
Ap [rad]
0.5
-0.1
00, [mmss]
12.5
[Nm]
200
10.0
100-
25
100
150
175
200
25
50
100
Distance along path [m]
Distance along path [m]
Figure 16: Lexus drifting first- gear donut trajectory with Tire 2.
Reference
Expert (Tires 2)
Ditfusion (Tires 2, 3)
BaseSDE (Tires 2, 3)
V (m/s)
10-
M
r/rd/6)
20
[deg)
5 [deg]
0.2
[rad]
ml
-02
14-
500
R ob, [m/s]
[Nm]
200-
10-
100
30
100
150
200
250
300
100
150
200
250
300
Distance along path [m]
Distance along path [m]
Figure 17: Lexus drifting first-gear Flgure-8 trajectory with Tire 2.
26

---

Reference Diffusion
mm
0.0
V [m/s)]
T [radls)
-1.0-
-1.5
B [deg]
5 [deg]
-20
de [rad]
e [m)]
0.0
-0.2-
R R-O,, [m/s]
12.5
150
rf [Nm]
100
10.0
7.5-
200
150
200
Distance along path [m]
Distance along path [m]
Figure 18: Lexus drifting on a first-gear donut trajectory on heavy rain.
27


Reviews:
[{"review_id": "iITeD4XcZh", "reviewer": "Reviewer_Xoo7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors designed and trained a diffusion model to map recent trajectory observations into parameter values of a neural SDE (stochastic differential equation) model, which is in turn used in MPC to control a car at the limits of handling (specifically tire dynamics). The trained diffusion model is shown to generalize across vehicle types, weather conditions, and reference trajectory geometry. The authors claim that their main contribution is a \"reliable method\" for autonmous driving \"at the limits of handling\".", "summary_of_recommendation": "The recommendation is based on the current state of the paper: it introduced a method with novelty to address a meaningful enough challenge (automated car control in static environment at the limits of handling) and demonstrated the effectiveness of the method. The real-world demonstration, especially that of the generalization across car type, weather condition, and trajectory shape, is compelling. The way in which a diffusion model is combined with neural SDE, and through which with MPC, is interesting. That this reviewer thinks the potential of the general architecture goes beyond the authors' own realization certainly adds to the reviewer's own confierence in their recommendation but does not change what's recommended, which again is based on the strength of the current manuscript itself.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Equation 1: \"$dB$\" should be \"$dW$\" to be consistent with Figure 1?\n\n2. Line 125 the variable $z$ seems to have never been introduced.\n\n3. Algorithm 1, Line 6: the condition \"if Time to refine\" seems to have never been explained.\n\n4. Algorithm 1, Line 10: \"Update D with ...\" should be something like \"Append ... to D\" instead? \"Update\" is too vague.\n\n5. Line 138: \"$2W$\" -- the variable $W$ was never introduced.\n\n6. Lines 145-146: more explanation?\n\n7. Lines 163-165: this seems quite standard in dealing with diffusion models. Should we just defer to the literature? Otherwise, if there's something new and special, explain!\n\n8. Line 248: \"This conclusion ...\" -- it's not advicable to do anaphoric reference to the previous section at the beginning of a new section.\n\n9. This reviewer disagrees with the \"meta-learning\" characterization (Line 86; see Review above for discussion). The authors are encouraged to consider their approach as hierarchical.\n\n10. The authors are also encouraged to discuss Algorithm 1, especially it's complexity from a hierarchical perspective.\n\n11. In terms of limitations, please also consider the method's relevance to handling general complexity of an interactive driving environment (with other road users). Now, in real world handling of driving in snow and ice, it's simply not true that the environment has only the ego car itself. This comment/request is of course also along the recommendation above of considering the method's general relevance.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4V9VtHWG57", "reviewer": "Reviewer_3p3m", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper deals with the challenge of operating cars on their limits, i.e. in drifting situations, which is critical for safe autonomous driving in critical situations. The authors propose to use a physics-informed SDE which parameters are learned by a diffusion model. The article shows very strong results in the experimental section where the proposed method is applied to two real-world cars.", "summary_of_recommendation": "Paper with strong experimental verification on operating cars in drifting situation which should be accepted at Corl.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Please address the issues mentioned in the “review” section", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j9c178wdYn", "reviewer": "Reviewer_ceJU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a generative model that can handle driving at the limits. A physics informed neural SDE model is introduced for this. Experiments show that the model can perform drifting maneuvers on 2 different vehicles", "summary_of_recommendation": "The paper is an interesting read with relevant results for the robotics community. However, there are aspects which are not very clear and require further clarficiation.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "In addition to the questions raised in the previous review section, there are some additional points that require explanation:\n\n1) What are the control inputs (L117) and where do they come from? If they are already available, would this not be considered labeled data?  Also, are the controls not the final output for eventually controlling the vehicle?  \n\n2) How fast is the online adaptation of the neural-SDE (L87)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a method to map recent trajectory observations into parameter values of a neural SDE model via a diffusion model, which is, in turn, used in MPC to control a car at its limits.

Strengths
- The paper presents an interesting approach that enables driving at the limit without access to labeled training data.
- The presented experiments on two different real-world cars are compelling.

Limitations
- The paper needs to provide further clarification on some of the technical details to fully understand its contributions, such as the availability of the control inputs during training, the latency of online adaptation of the neural SDE, etc.
- It would also be helpful to comment on how the method would handle interactive driving scenarios.

**Comments post rebuttal**

The revised paper has satisfactorily addressed the outstanding reviewers' concerns/questions.
Decision: Accept