A Diffusion-Model of Joint Interactive Navigation
Authors: Matthew Niedoba
Jonathan Wilder Lavington
Yunpeng Liu
Vasileios Lioutas
Justice Sefas
Xiaoxuan Liang
Dylan Green
Setareh Dabiri
Berend Zwartsenberg
Adam Scibior
Frank Wood
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Models
Trajecotry Forecasting
Autonomous Vehicles
Motion Forecasting
Simulation
TL;DR: We present a diffusion model which generates joint traffic scenes, and investigate its performance and test time conditioning methods.

Abstract: Simulation of autonomous vehicle systems requires that simulated traffic participants exhibit diverse and realistic behaviors. The use of prerecorded real-world traffic scenarios in simulation ensures realism but the rarity of safety critical events makes large scale collection of driving scenarios expensive. In this paper, we present DJINN -- a diffusion based method of generating traffic scenarios. Our approach jointly diffuses the trajectories of all agents, conditioned on a flexible set of state observations from the past, present, or future. On popular trajectory forecasting datasets, we report state of the art performance on joint trajectory metrics. In addition, we demonstrate how DJINN flexibly enables direct test-time sampling from a variety of valuable conditional distributions including goal-based sampling, behavior-class sampling, and scenario editing.

A Diffusion-Model of Joint Interactive Navigation
Matthew Niedoba!
J. Wilder Lavington"
Yunpeng Liu
Vasileios Lioutas
Justice Sefas1.2
Xiaoxuan Liang Dylan Green
Setareh Dabiri²
Berend Zwartsenberg' Adam Scibior Frank Wood
University of British Columbia, Inverted Al
mmiedobaêcs. .uc. ca
Abstract
Simulation autonomous vehicle systems requires that simulated traffic partici- pants exhibit diverse and realistic behaviors. The use of prerecorded real-world traffic scenarios in simulation ensures realism but the rarity of safety critical events makes large scale collection of driving scenarios expensive. In this paper, present DJINN a diffusion based method of gencrating traffic scenarios. Our approach jointly diffuses the trajectories of all agents, conditioned on flexible set of state observations from the past, present, or future. On popular trajectory forecasting datasets, metrics. In addition, we demonstrate how DJINN flexibly enables direct test-time sampling from variety of valuable conditional distributions including based sampling, behavior-class sampling, and scenario editing.
report state of the art performance joint trajectory
Introduction
Accurate simulations are critical to the development of autonomous vehicles (AVs) because they facilitate the safe testing of complex driving systems [15]. One of the most popular methods of simulation is virtual replay [46], in which the performance autonomous systems are evaluated by replaying previously recorded traffic scenarios. Although virtual replay is valuable tool for AV testing, recording diverse scenarios is expensive and time consuming, as safety-critical traffic behaviors rare [17]. Methods for producing synthetic traffic scenarios of specific driving behaviors are therefore essential to accelcrate AV development and simulation quality.
Producing these synthetic traffic scenarios involves generating the joint future motion of all the agents scene, task which closely related to the problem of trajectory forecasting. Due to the complexity of learning fully autonomous end-to-ent vehicle controller, researchers often opt to split the problem into three main tasks [52]] perception, trajectory forecasting, and planning. In trajectory forecasting, the future positions of all agents predicted up to specified future time based on the agent histories and the road information. Due to the utility of trajectory forecasting models in autonomous vehicle systems along with the availability of standard datasets and benchmarks to measure progress [4, 53], a variety of effective trajectory forecasting methods are now available. Unfortunately, most methods produce deterministic which are difficult to combine to produce realistic joint traflic scenes [30].
of trajectory forecasts per-agent [47, 9]
Generative models of driving behavior have been proposed an alternative to deterministic trajectory forecasting methods for traffic scene generation [40, 46). These models re- frame trajectory forecasting past observations and map
modeling the joint distribution of future agent state conditioned context. However, given that the distribution of traffic scenes in motion forecasting datasets are similar to real-world driving, modelling the data distribution does not ensure that models will generate rare, safety critical events.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To alleviate these issues we propose DJINN, a model which generatively produces joint traffic scenarios with flexible conditioning. DJINN diffusion model over the joint states of all agents in the scene. Similar to [30]. our model is conditioned on flexible set of agent states. By modifying the conditioning set at test-time, DJINN is able to draw traffic scenarios from a variety of conditional distributions of interest. These distributions include sampling scenes conditioned on specific goal states or upsampling trajectories from sparse waypoints. Additionally, the joint diffusion structure of DJINN enables test-time diffusion guidance. Utilizing these methods enables further control over the conditioning of traffic scenes based on behavior modes, agent states, or scene editing.
We evaluate the quality of sampled trajectories with both joint and ego-only motion forecasting on the Argoverse [4] and INTERACTION [53] datasets. We report excellent ego-only motion forecasting and outperform Scene Transformer on joint motion forecasting metrics. We further demonstrate both DJINN's s flexibility and compatibility with various forms of test-time diffusion guidance by generating goal-directe samples, examples of cut-in driving behaviors, and editing replay logs.
2 Related Work
Trajectory Forecasting: A wide variety of methods have been proposed to address the problem of trajectory forecasting. Two attributes which divide this area of work are the output representation type and the agents for which predictions are made. The most common class of models deterministically predict the distribution of ego agent trajectories using weighted trajectory set either with without uncertainties. Due to the applicability of this representation as the input for real-time self-driving planners, there are numerous prior methods of this type. Some approaches rasterize the scene into birdview image and use CNNs to predict a discrete set of future trajectories for the ego agent [5. 3, 32]. The convolutional architecture of these methods captures local information around the agent well, but the birdview image size and resolution limit the ability to capture high speed and long-range interactions. To address these challenges, other prior approaches encode agent states directly either by using RNNs [47, 39, 27]. polyline encoders [7, 9] or ID convolutions [24]. Agent features can be combined with roadgraph information in a variety of ways including graph convolutional networks [24, 1] or attention [29, 27].
To control the distribution of predicted trajectories, several methods have utilized mode goal conditioning. One approach is to directly predict several goal targets before regressing trajectories to those targets [54, 9, 51, 8). An alternate approach is to condition on trajectory prototypes [3] or latent embeddings [47].
Predicting joint traffic scenes using per-agent marginal trajectory sets is challenging due to the exponential growth of trajectory combinations. Recent approaches aim to rectify this by producing joint weighted sets of trajectories for all agents in scene. M21 [45] generates joint trajectory sets by producing "reactor" trajectories which are conditioned on marginal "influencer" trajectories. Scene Transformer [30], which uses similar backbone architecture to our method, uses transformer [48] network to jointly produce trajectory sets for all agents in the scene.
As an alternative to deterministic predictions, multiple methods propose generative models of agent trajectories. variety of generative model classes have been employed including Normalizing Flows [36). GANs [10, 38] or CVRNNs [40, 46). Joint generative behavior models can either produce entire scenarios in one shot [10, 38, 36), or produce scenarios by autoregressively "rolling-out" agent trajectories [40, 46].
Diffusion models, proposed by Sohl-Dickstein et al. [41] and improved by Ho et al. [12] are class of generative models which approximate the data distribution by reversing a forward process which gradually adds noise the data. The schedule of noise addition can be discrete process or represented by a continuous time differential equation [44, 21 We utilize the diffusion parameterization introduced in EDM [21] in our work for its excellent performance and separation of training and sampling procedures.
Diff'usion Models:
This class of models have shown excellent sample quality in a variety of domains including images [12. 6], video [11, 14] and audio [22]. In addition, diffusion models can be adapted at test-time through various conditioning mechanisms. Classifier [6] and classifier-free guidance [13] have enabled powerful conditional generative models such as text conditional image models [34, 37] while editing techniques [26, 31] have enabled iterative refinement of generated samples.

---

Predictive
Goal Conditioning
Agent Reactive
Up-Sampling
Impuation
Time
observed
generated
Figure Top:
Five example observation masks O demonstrating potential conditioning inputs to DJINN. Each element of cach mask corresponds to the boolean value of O for that agent state. Individual agents are shown in rows, with timesteps as columns. Bottom: Generated traffic scenes
corresponding to the type of observation masks above.
One recent application of diffusion models is planning. Diffuser [20] uses diffusion models to generate trajectories for offline reinforcement learning tasks. They condition their samples using classifier guidance to achieve high rewards and satisfy constraints. Trace and Pace [35] utilizes diffusion planning for guided pedestrian motion planning. In the vehicle planning domain, Controllable Traffic Generation (CTG) [55] builds on Diffuser, using diffusion models to generate trajectories which satisfy road rule constraints. Like CTG, method also models the future trajectories of road users using diffusion models. However, our approach differs from CTG in terms of output and our methods of conditioning. CTG, marginal per-agent trajectory samples representation by "rolling-out" a portion of each agent's trajectory before drawing new samples per-agent. By contrast, DJINN models the full joint distribution of agent trajectories in one shot, with re-plamning or roll-outs required. The authors of CTG condition their model exclusively on the past states of other agents and the map, and use classifier classifirr-uiddance to follow road rules. In our method, we demonstrate conditioning on scene semantics via classifier guidance as well as conditioning on arbitrary state observations, including the past future states of each agent, and control the strength of conditioning using classifier-frs guidance as demonstrated in Fig.
combined into joint scene
condition their samples
Background
3.1 Problem Formulation
Our work considers traffic scenarios consisting of agents across discrete timesteps driving on roadway described by set of roadgraph features M. Each agent E {1....A) A} in the scene at time E [1,... T} is represented by state st = {xt, yf 0 consisting of its 2D position (x,, y1) and heading 9,. The joint representation of the scene is the combination of all agents across all timesteps x {s la E (1,..., A}. according to unknown distribution Paata(x)) (,....,T) E TAxTx3 We assume scenes are distributed
We introduce model which conditioned on the map features M and can moreover be flexibly conditioned on arbitrary set of observed agent states. For the latter purpose, we consider boolean variable
(0, 1 AXT_ We denote that state in the scene observed if O 1. Using O, we partition the scene into two components. The observed portion of the scene is defined as ) Xobs Figure shows five choices for O and their corresponding tasks. Our ultimate goal is to learn a conditional distribution over the set of all latent agent states Xiat given the observed states Xobs and the map M. by modelling p( p(xtadla Xobe, M). Using this probabilistic framework,
Xobs = {s] ss E x, Og = }} while the unobserved, latent portion is Xiat
represent conditional

---

distributions corresponding to various trajectory forecasting tasks by modifying the observation mask O and the corresponding conditioning set Xohs-
3.2 Diffusion Models
Diffusion models [41, 12] are powerful class of generative models built upon a diffusion process which iteratively adds noise the data. In the continuous time formulation of this process [44, 21], this iterative addition is described by stochastic differential equation (SDE)
dx = p( 7)dT (r)
Here, T E [0, Tmaz] where Tmaa fixed, large constant, ,u(x,, 7) is the drift function and (T() is the diffusion coefficient which scales standard Brownian motion w. Note that our work has two notions of time. Throughout we will use to denote the "scenario time" and We express the marginal distribution of at diffusion time T as p(x+), with p(xo) corresponding to the data distribution Pdata (x). Typically, (x,, T), (T)) and Tmar are chosen such the conditional density p(x+(xo) is available in closed form and that p(X approximates tractable Gaussian distribution (x). Notably, for every diffusion SDE, there exists a corresponding probability flow (PF) ordinary differential equation (ODE) [44] whose marginal probability densities p(x-) match the densities of Eq. (1)
represent "diffusion time".
1 dx, = ((xx,T) T) 2 (T)
DerVvlogpsss log p(x,) ) dr.
(2)
Using the PF ODE, samples are generated from diffusion model by integrating Eq. (2) from Tmaz to r = O with initial condition Ximaz ~ π <xXxmaz) using an ODE solver. Typically integration is stopped some small value for numerical stability. Solving this initial value problem requires evaluation of the score function Vx, log ((x)( Since p(x+) is not known in closed form, diffusion models learn approximation of the score function sp(X,, T) Vx, log p(x,) via score matching [16, 43, 44].
A useful property of diffusion models is the ability to model conditional distributions p(Xoly) at test-time using guidance. Given some conditional information y. the key idea of guidance is to replace the score function in the PF ODE with approximate conditional score function Vx, log p(x, ly).
By using the gradient of a pretrained classifier Po xx), glassifier guidance [6] approximates the conditional score function through the linear combination of the unconditional score function and the classifier gradient. The parameter a controls the strength of the guidance
Vx, log p(x,11) ly) R (x T) + a Vxx log po(y)))))
One major drawback of classifier guidance is the need to train an external classifier. Instead, classifier- free guidance [13], utilizes a conditional score network sy (x,, r, y). Then, weighted average the conditional and unconditional scores is used to estimate the conditional score function.
V log p(x+ly) Ri Xso T, + ()so((+:
(4)
Here À is scalar parameter which controls the strength of the guidance. In both cases, the approxi- mate conditional score can be substituted into Eq. (2) to draw conditional samples from p(Xoly).
DJINN
Our approach models the joint distribution agent states p(Xtat Xohs, M) conditioned on a set of observed states and the map context. For this purpose, we employ a diffusion model which diffuses directly over Xtat the unobserved states of each agent in the scene for f = (1....T) An important aspect of our method is the choice of observation mask O and observation set Xode on which we condition. For this purpose introduce a distribution over observation masks p(O) which controls the tasks on which we train our model.
In the design of our diffusion process, we follow the choices from EDM [21], setting (Xiitt, 7) 0 and ((r) 22T from Eq. (2). We also utilize their score function parameterization

---

Xobs: M, c) Do (Klat.r, Xobs, M. T) Xjat,r
(S)
Kiat.r
x2
Here D9 is a neural network which approximates the latent portion of the noise freee data Xlat,- In addition to Xiat,r and т, in work D3 also receives the map context M, the clean observed states Xobs and c, a collection of unmodelled agent features per observed agent timestep such as velocity, vehicle size, agent type. We train network modification of the objective from EDM [21]
8x0,7, Ntass, || Do (Xtat,r Xobs: M.C. C. r) Xlat,ol1
Here, Xo Pdata (x), x, p(,, xn) NVxx 2) and O p(O). We compute loss over ~e Ptraén - log normal distribution which controls the variance of the noise added to the data. We set the mean and variance of Ptrain according [21].
We use the Heun 2nd order sampler from [21] to sample traffic scenarios with no changes to the reported hyperparameters. Empirically, we found that deterministic sampling, corresponding to integrating the PF ODE, leads to higher quality samples than using an SDE solver. Unless otherwise noted all samples are produced using 50 iterations of the ODE solver, which produces the highest quality samples as measured by ego and joint minADE and minFDE.
Input Representation An important choice for trajectory forecasting models is the reference frame for the agent states. In our work, the diffused agent states and observations Xoba are centered around an "ego agent," which is often specified in trajectory forecasting datasets as the primary agent of interest. We transform Xg such that the scene is centered on the last observed position of this arbitrary "ego agent" and rotated so the last observed heading of the ego agent is zero. We scale the positions and headings of all agents in each ego-trans gromfrranff scene to a standard deviation of 0.3.
We represent the map M as an unordered collection of polylines representing the center of each lane. Polylines are comprised of a fixed number of 2D points. We split longer polylines split into multiple segments and pad shorter polylines padded to the fixed length. Each point has a boolean variable indicating whether the element padding. Polyline points are represented the same reference frame as the agent states and are scaled by the same amount as the agent position features.
Model Architecture Our score estimator network Dg parameterized by transformer- based architecture similar to [30]. The network operates on a fixed [A, F" shaped feature tensor composed of one F dimensional feature vector per agent timestep. We use sinusoidal positional embeddings [48] to produce initial feature tensors. Noisy and observed agent states X,, Xobs. the time indices t= (1..... T). and diffusion step 7 are all embedded into F dimensional embeddings. Xiat.r and Xobs are padded with zeros for observed and latent states respectively prior to embedding. A shared MLP projects the concatenated positional embeddings into dimensional vector for each agent.
The main trunk of the network is comprised of a series of transformer layers [48]. Attention between all pairs of feature vectors factorized into alternating time and agent transformer layers. In time transformer layers, self-attention is performed per-agent across each timestep of that agent's trajectory, allowing for temporal consistency along a trajectory. In agent transformer layers, self-attention is computed across all agents at given time, updating each agent's features with information about the other agents at that time. We encode the map information M with a shared MLP that consumes flattened per-point and per-lane features to produce fixed size embedding per lane. Cross attention between the collection of lane embeddings and agent states incorporates map information into the agent state features. Our network is comprised of 15 total transformer layers with fixed feature dimension of 256. We use an MI.P decoder after the final transformer layer to produce our estimate of Xiat,- A full representation of our architecture is available in Appendix A.
5 Guidance for Conditional Scene Generation
So far, we have outlined our method for generating joint traffic scenes using DJINN. Next, we describe how the diffusion nature of DJINN enables fine-grained control over the generation and modification of driving scenarios.
5

---

Table 1: Ego-only motion forecasting performance Argoverse and INTERACTION datasets.
minADE and minFDE metrics on both datasets indicate that DJINN produces ego samples which closesly match the distribution of ego agent trajectories.
(a) Argoverse test set
(b) INTERACTION validation set
Method
minADEg
minFDEg
Method
minADEg
minFDEg
Jean [27]
0.98
1.42
DESIRE [23]
0.32
0.88
mmTransformer [25]
0.87
1.34
TNT [54]
0.21
0.67
TN' [9]
0.88
1.28
ReCoG [28]
0.19
0.65
MultiPath++[47]
0.79
1.214
ITRA [40]
0.17
0.49
DCMS [50]
0.77
1.14
StarNet [19]
0.13
0.38
AceneTrrasformm [30]
0.80
1.23
SAN [18]
0.10
0.29
Ours
1.02
1.65
Ours
0.14
0.39
Table Ego-only and joint metrics comparing DJINN to a jointly trained Scene Transformer model
on the Argoverse validation set. DJINN produces better joint samples than Scene' Transformrr when measured by minSceneADE and minSceneFDE.
Method
minADEg
minFDEg
minScene
minSceneFDEs
Scene Transformer (Joint)
0.848
1.398
1.019
1.835
Ours
0.871
1.409
0.895
1.758
5.1 Classifier-free Guidance
In Scene Transformer [30], masked sequence modelling framework is introduced for goal-directed and agent-reactive scene predictions. One limitation of this approach that conditioning performed on precise agent states while future agent states or goals are usually uncertain. We mitigate this limitation through the use of classifier guidance.
We assume access to a set of precise observations Xahs, and some set of additional agent states Xoond sample. For instance, Xcond may include agent goals upon which
which we wish condition we wish to condition. Let X (Xobs U Xcond) J. Based on Eq. (4), the conditional score is through weighted average of the score estimate conditioned Xoba and the estimated conditioned on x
Xist. log p(xiet, Xabs) xd
D9 (Xiat, X M. M.C.T) T)
Xiat.r
(1 A) Do (xtat,r Xabs, M, M,C,T) Xiot,r
To facilitate classifier-free free conditioning, we train DJINN on a p(O) representing varied conditioning tasks. These tasks include conditioning agent history, agent goals, windows of agent states, and random agent states. A full overview of our task distribution is given in Appendix B.
5.2 Classifier Guidance
Many driving behaviors of individual multiple agents can be categorized by class y based on their geometry, inter-agent interactions or map context. Examples of classes include driving maneuvers such as left turns, multi agent behaviors such as yielding to another agent, constraints such as trajectories which follow the speed limit. DJINN uses classifier guidance to conditioned scenes these behavior classes. Given set of example scenes corresponding y. we train a classifier to model (y(x).. Using Eq. (3) we approximate the conditional score for conditional sampling. Importantly, due to the joint nature of our representation, classifiers for per-agent, multi-agent or whole-scene behaviors can be all used to condition sampled traffic scenes.
behavior class
5.3 Scenario Editing
One benefit of sampling traffic scenes at once instead of autoregressively is the ability to edit generated or recorded traffic scenarios through stochastic differential editing [26]. Given a traffic scene x, a user manually modify the trajectories in the scene to produce "guide" scene x' which approximates

---

A=0.1
A=0.3
A=10
abserved
generated
*
goal
Figure The effect of classifier free guidance weight on the spread trajectories for goal conditioned sampling. Samples drawn from the INTERACTION validation set conditioned using classifier free guidance on goal state (star). As the guidance weight increases, deviation from the goals decreases.
the desired trajectories in the scene. The guide scene is used to condition the start of truncated reverse diffusion process by sampling X ~ NN'x,, Toda where Tedit is an intermediate time in the diffusion process between and Tman- Then, the edited scene is produced by integrating the PF ODE using the same ODE solver, starting from initial condition Todit- Through the stochastic differential editing, the guide scene is translated into realistic traffic scene with agent trajectories which approximate the guide trajectories. We empirically find Tedit = 0.8 to be a good trade-off between generating realistic trajectory scenes and maintaining the information of the guide scene.
Experiments
6.1 Motion Forecasting Performance
To measure the quality of the samples from DJINN, we evaluate our method on two popular motion prediction datasets, matching O during training to match each dataset. For the INTERACTION dataset [53] scenes, we observe the state of all agents over the first second of the scene and generate the next three seconds. On the Argoverse dataset [4] our model observes agent states over the first two seconds of the scene and generates the next three seconds. Training hyperparameters for both models are found in Appendix A.
We note that both INTERACTION and Argoverse metrics measure an ego-only trajectory-set using minADE and minFDE over 6 trajectories. Since DJINN produces stochastic samples of entire traffic scenes, a set of random trajectories may not cover all future trajectory modes. To alleviate this, we draw collection of 60 samples for each scenario and fit 6 component Gaussian mixture model with diagonal covariances using EM in a method similar to [47]. We use the means of the mixture components as the final DJINN prediction for motion forecasting benchmarks.
We present DJINN's performance on motion forecasting Table with Argoverse results in Table la and INTERACTION results in Table 1b. On INTERACTION, DJINN generates excellent ego vehicle trajectories, with similar minFDE and minADE to state of the methods this dataset. On the Argoverse test set we produce competitive metrics, although our results lag slightly behind top motion forecasting methods. We hypothesize that our lower performance on Argoverse due the lower quality agent tracks in this dataset when compared to INTERACTION.
We further analyze the joint motion forecasting performance of DJINN. To this end, measure the Scene minADE and minFDE proposed by [2) which measures joint motion forecasting performance over collection of traffic scenes. We compare DJINN against a reproduction of Scene Transformer trained for joint motion forecasting, using their reported hyperparameters. Ego-only and Scene motion forecasting performance is shown in Table 2. Although Scene Transformer predicts slightly better ego vehicle trajectories, we demonstrate DJINN has superior joint motion forecasting capabilities.
6.2 State-conditioned Traffic Scene Generation
While DJINN is able to draw samples for motion forecasting benchmarks by conditioning past observations the scene, key benefit of our approach the ability to flexibly condition at test-time
7

---

Scene
Scene
Ungulded
Gulded
Gulded
Figure 3: Examples of synthetic cut-in behaviors generated using classifier guidance. Samples generated from the INTERACTION validation set conditioned on the first 10 agent states. Applying classifier guidance causes the other agent (green) cut in front of the ego agent (purple). We generate trajectories for all agents in the scene, but other agent trajectories have been omitted for clarity
based on arbitrary agent states. We illustrate this test-time conditioning in Fig. by generating samples from five conditional distributions which correspond to use-cases for our model.
Specifying exact agent states on which condition can be challenging. One approach is to utilize the states of prerecorded trajectory to produce conditioning inputs. However, if one wishes to generate trajectory which deviates from recorded trajectory, there uncertainty about the exact states on which to condition. In Fig. 2, we demonstrate how classifier-free guidance can be utilized to handle user uncertainty in conditioning agent states. In this example, we the observation set Xoòs to the first ten states of each agent's recorded trajectory. Further, we create a conditional observation set Xoto by augmenting Xobs with goal state for each agent drawn from normal distribution centered on the ground-truth final position of each agent, with m variance. We sample traffic scenes with varying levels of classifier-free guidance strength, drawing two conclusions. First, DJINN robust to goals which do not match the recorded final agent states. Secondly, the strength of the classifier guidance weight controls the emphasis of the goal conditioning, resulting in trajectory samples which cluster more tightly around the specified goal the guidance strength is increased. With low guidance weight, the samples are diverse, and do not closely match the specified goal position. As the weight increases, the spread of the trajectory distribution tightens, especially for fast, longer trajectories. These properties give users finer control over the distribution of traffic scenes when there is uncertainty over the conditioning states.
6.3 Conditional Generation from Behavior Classes
We now continue to demonstrate the flexibility of our approach by considering test-time conditioning of our model on specific driving behaviors through classifier guidance. Specifically, we highlight the ability to condition DJINN on the behavior class of cut-in trajectories by conditioning our INTERACTION trained model with cut-in classifier.
A "cut-in" occurs when one vehicle merges into the path of another, often requiring intervention by the cut-off driver. We selected this behavior demonstrate how classifier guidance can be used with our joint representation to sample scenes conditioned on the behavior of multiple agents. We condition DJINN trained on INTERACTION using simple cut-in classifier. To train the classifier, we first mined a dataset of cut-in behaviors trajectory pairs from the "DR_ BR__CHN_Merginng _er_MMrrnnn___ _z" location highway driving scene with some cut-in examples. Each trajectory pair comprised of "ego" and an "other" agent. We define positive cut-in as case where the future state of the other agent at time lother overlaps with a future state of the ego agent time lego such that Lego 3s < lother < lego- Further, we filter cases where the initial state of the other agent overlaps with any part of the ego

---

Scene
Original Scene
Manually Modified Scene
Generated Scene
Scene
observed
generated
manual guide
Two scenario fine-tuning examples (one per row) based on Argoverse validation set scenarios. Left: original scene with ground-truth trajectories shown for two interacting vehicles, vehicle positions at the same time index for all agents. Middle: manual edit of one agent's 's trajectory each scene. One (top) replaces a right turn with forward continuation, the other (bottom) shifts a trajectory back in space to cause complex interaction to occur near the end of the trajectory. Right: the resulting stochastic differential edit of the original scenario. Both rows of the last column illustrate joint reactivity to the new trajectories arising from the edit; in the top row the left-turning vehicle yields and in the bottom row both trajectories shift avoid collision.
Figure 4:
trajectory to eliminate lane following cases. We label negative cut-in trajectories in which the minimum distance between any pair of ego and other states is less than 5m.
any other pair of
Using these heuristics, we collect dataset of 2013 positive and 296751 negative examples. We trained a two layer MLP classifier with 128 dimensions per hidden layer. The classifier takes as input the diffused trajectories of each agent, the validity of each timestep and the diffusion time Using this classifier, we generate synthetic cut-in scenarios Eq. (3). Examples our synthetic cut-in scenarios are found in Fig. 3. The generated scenarios clearly demonstrate our model can be conditioned to create synthetic cut-in behaviors. These synthetic examples provide evidence that given a collection of trajectories exemplifying behavior mode, heuristic which can be used to generate example trajectories, DJINN can conditioned to generate synthetic examples representing that behavior mode. This finding further expands the flexibility of our model to generate trajectory samples from valuable conditional distributions.
6.4 Scenario Tine-Tuning
We exhibit another method of controlling the traffic scenarios generated with DJINN through fine- tuning. Since DJINN diffuses entire traffic scenes without iterative replanning, we are able to use stochastic differential editing to modify the sampled scenes. Given a recorded sampled traffic scene, differential stochastic editing can used to fine-tune the scene through the use a manually specified guide. In Fig. new scenarios with realistic trajectories but complex interactions. Using two recorded validation set scenes from Argoverse,
demonstrate how DJINN can fine-tune existing scenarios to produce
aim to edit the scenes to generate more interactive trajectories between the agents. For this purpose, we generate an guide scene Xguide by manually adjusting the trajectories in each scene that the future paths of two of the agents will intersect. Through stochastic differential editing, we show that DJINN is able to produce realistic driving scenes which shift the guide scene trajectories maintain their interactivity but avoid collisions between agents.

---

7 Conclusions
In this work, we present DJINN diffusion model of joint traffic scenes. By diffusing in a joint agent state representation, DJINN can be adapted at test time to a variety of modeling tasks through guidance methods and scenario editing. The power of this scenario generation model opens exciting possibilities. Future research may expand the variety of guidance classifiers such as utilizing the classifiers proposed in [55] for traffic-rule constraint satisfaction. Another promising avenue of research is scaling DJINN for faster scenario generation. Although flexible, the diffusion structure of DJINN makes scenario generation relatively slow due the iterative estimation of the score function. Distillation techniques such as consistency models [42] may be helpful in this regard to improve the number of score estimates required per sample. Future work may also consider scaling the length and agent count in generated scenarios to improve the complexity of behaviors which can be generated. Other areas of future work include using DJINN in a model predictive control setting (hinted at the predictive mask of Fig. 1) in which an ego action is scored using statistics of ego-action conditioned joint trajectories from DJINN.
Acknowledgements
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada CIFAR AT Chairs Program, Inverted AI, MITACS, the Department of Energy through Lawrence Berkeley National Laboratory, and Google. This research was enabled in part by technical support and computational resources provided by the Digital Research Alliance of Canada Compute Canada (alliancecan.ca ca), the Advanced Research Computing at the University of British Columbia (arc.ubc.ca), Amazon, and Oracle.
References
[1] Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urtasun. Spagnn: Spatially-aware graph neural networks for relational behavior forecasting from sensor data. In 2020 IEEF International Conference on
Robotics and Automation (ICRA), pages 9491-9497. IEFF, 2020.
[2] Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In Computer Vision-ECCV 2020: 16th European Conference, Glargow, UK, August 23-28, 2020. Proceedings, Part XXIII 16, pages 624-641 Springer, 2020.
[3] Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In Conference on Robot Learning, pages 86-99.
PMLR, 2020.
[4] Ming-Fang -ang Chang. John Lambert, Patsorn Sangkloy, Jagject Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey. Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8748-8757, 2019.
[5] Henggang Cui, Vladan Radosavljevic, Fang- Frang-Chieh Chou, Tsung- Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 International Conference on Robotics and Automation (ICRA), pages
2090-2096. IEEE, 2019.
[6] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in
Neural Information Processing Systems, 34:8780- 0794,, 2021.
[7] Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectomett: Encoding hd maps and agent dynamics from vectorized representation. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11525 11533, 2020.
[8] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Gohome: Graph-oriente heatmap output for future motion estimation. In 2022 International Conference Robotics and Automation (ICRA). pages 9107-9114, 14. IEEE, 2022.
[9] Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15303- 15312, 2021. [10] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 2255-2264, 2018
[1]] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Dietrich Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. In Advances in Neural Information Processing Systems, 2022.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-655 18500066111100 2020.
10

---

[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[14] Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet.
Video diffusion models. In Alice I1. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors. Advances in Neural Information Processing Systems, 2022.
[15] WuLing Huang, Kunfeng Wang. Yisheng Lv, and FengHua Zhu. Autonomous vehicles testing methods review. In 2016 IEEE/ 19th International Conference on Intelligent Transportation Systems (ITSC), pages
163-168. IEEE, 2016.
[16] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[17] Ashesh Jain, Luca Del Pero, Hugo Grimmett, and Peter Ondruska. Autonomy 2.0: Why is self-driving
always 5 years away? arXiv preprint arXiv:2107.0814: 2021.
[18] Faris Janjoš, Maxim Dolgov, Muhamed Kurić, Yinzhe Shen, and J Marius Zöllner. San: Scene anchor networks for joint action-space prediction. In 2022 IEEE Intelligent Vehicles Symposium (IV), pages
1751-1756. IEEE, 2022.
[19] Faris Janjoš. Maxim Dolgov, and J Marius Zöllner. Starnet: Joint action-space prediction with star graphs and implicit global frame self-attention. arXiv preprint arXiv:2111. 2021.
[20] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffussion for flexible behavior synthesis. In International Conference on Muchine Learning, 2022.
[21] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of feflusioo-based
generative models. In Advances in Neural Information Processing Systems, 2022.
[22] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion
model for audio synthesis. In International Conference on Learning Representations, 2021.
[23] Namhoon Lee, Wongun Choi, Paul Vernza,, Christopher B Choy, Philip HS Torr, and Manmohan Chan- draker. Desire: Distant future prediction in dynamic seenes with interacting agents. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pages 336-345, 2017.
[24] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lanc graph representations for motion forecasting, In European Conference Computer Vision, pages 541-556. Springer, 2020.
[25] Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7577-7586, 2021.
[26] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on
Learning Representations, 2021.
[27] Jean Mercat, Thomas Gilles, Nicole EI Zoghby, Guillaume Sandou, Dominique Beauvois, and
Guillermo Pita Gil. Multi-head attention for multi-modal joint vehicle motion forecasting. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9638 9644. IEEE, 2020.
[28] Xiaoyu Mo, Yang Xing, and Chen Lv. Recog: A deep learning framework with hetcrogeneous graph for interaction-awart trajectory prediction. arXiv preprint arXiv:2012.00 050032, 2020.
[29] Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled Refaat, and Benjamin Sapp.
Wayformer: Motion forecasting via simple & efficient attention networks. arXiv preprint arXiv:2207.0584.
2022.
[30] Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rehecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David Weiss, Ben Sapp. Zhifeng Chen, and Jonathon Shlens. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In International Conference on Learning Representations, 2022.
[31] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun Yan Zhu. Zero-shot image-to to im-i translation. arXiv preprins XxXiv:2302.03: 03002. 2023.
[32] Tung Phan Minh, Elena Corina Grigore, Freddy A Boulton, Oscar Beijbom, and Eric M Wolff Covernet: Multimodal behavior prediction using trajectory sets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14074-1408. 2020.
[33] Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[34] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional
image generation with clip latents. arXiv preprint arXi::2204.061225 2022.
[35] Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13756-13766 2023.
[36] Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction conditioned on goals in visual mulli--gent settings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2821 22302 2019.

---

[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion image synthesis with latent diffussin models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[38] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose. Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1349-1336 1358, 2019.
[39] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++:: Dynamically- feasible trajectory forecasting with heterogeneous data. In Computer Vision-ECCV ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XVIII 16, pages 683-700. Springer, 2020.
[40] Adam Scibior. Vasileios Lioutas. Daniele Reda, Peyman Bateni, and Frank Wood. Imagining the road ahead: Multi-agent trajectory prediction via differentiable simulation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 720-725. IEEE, 2021.
[41] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaaaaaahan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages
2256-2265. PMLR, 2015.
[42] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint 3Xiv:2333.01466 2023.
[43] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[44] Yang Song, Jascha Sohl- Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. accre-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[45] Qiao Sun, Xin Huang. Junru Gu, Brian C Williams, and Hang Zhao. M2i: From factored marginal trajectory prediction to interactive prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages -6452, 2022.
[46] Simon Suo. Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10400-10409, 2021.
[47] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Comman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguclov, et al. Multipath++: Efficient information fusion and trajectory aggregation for bchavior prodiction. In 2022 International Conference on Robotics and Automation (ICRA), pages 7814-7821. IEEE, 2022.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems,
30, 2017.
[49] Ruibin Xiong, Yunchang Yang. Di Hc, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan
Lan, Liwei Wang. and Ticyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pages 10524-10533. PMLR, 2020.
[50] Maosheng Yc, Jiamiao Xu, Xunnong Xu, Tongyi Cao, and Qifeng Chen. Dems:
Motion forecasting with dual consistency and multi-pseudo-targee supervision. arXiv preprint arXiv:2204.0883 2022.
[51] Wenyuan Zong, Ming Liang, Renjie Liao, and Raquel Urtasun. Lanerenn: Distributed representations for graph-centric motion forecasting. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 532-539. IEBE, 2021.
[52] Wenyuan Zeng, Wenjic Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8660-8669, 2019.
[53] Wei Zhan, Liting Sun, Wang. Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kummerle, Hendrik Konigshof, Christoph Stiller, Arnaud de La Fortelle, et al. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. arXiv 2019.
preprint
[54] Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp. Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. In Conference on Robot Learning, pages 895 904. PMLR, 2021.
[55] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. arXiv preprint 2Xiv:2210.17366
2022.
12

---

Appendix
A Model Details
A.1 Preconditioning
Following EDM [21], we precondition De by combining Xtat,r and the output of our network Fe using scaling factors
Dx(Xtat,7: Xobs: M, r) Cskip(T)Xiat; + Cout Fo Xabs: M, Cnoise (T)). (8)
We use the scaling values reported in [21] without modification but report them in Table 3 for convenience.
Table 3: Scaling Functions for Preconditioning
Scaling Factor
Function
Cskip
dataa (7 + 0 datal
Cout
T dato
2 + data
Cin
T2 + dataa
Choise
II((T)
Here, deta is the standard deviation of the diffusion features, We scale the positions and headings of the agent that data is 0.5 for all diffusion features.
A.2 Architecture
DJINN utilizes a transformer based architecture for Fg. An overview of the model structure is shown
in Fig. 5.
Feature Encoding
We encode the observed agent states Xobs, the noisy latent states Xlat,7, the scenario time and the diffusion time using sinusoidal positional encoding [48]. We represent the scenario time integer index increasing from 0 to T with corresponding to the earliest agent states. For each of the encoded features, produce a 256-dimensional encoding vector. An important hyperparameter for sinusoidal positional embeddings are the maximum and minimum encoding periods which we report in Table 4.
Table 4: Maximum and minimum positional encoding periods for DJINN input features
Feature
Minimum Period Maximum Period
Xobs Xlat,r
0.01
10
100
0.1
10,000
The concatenation of the positional encodings with additional agent state features fed through an MLP to form the input to the main transformer network. The additional agent state features consist of the agent velocity, the observed mask, and the agent size which is available for INTERACTION only. The input MI. is shared across all agent states and contains two linear layers with hidden dimension 256 and ReLU non- non-lincarities
Roadgraph Encoding
DJINN is conditioned on the geometry of the roadgraph through collection of lane center polylines. Each polyline is comprised of an ordered series of 2D points which represent the approximate center of each driving lane. We fix the length of each polyline to 10 points. We split polylines longer than this threshold into approximately equal segments, and pad shorter polylines with zeros. We utilize a boolean feature to indicate which polyline points are padded. Unlike (30), we do not use PointNet [33] to encode the roadgraph polylines. Instead, we encode the polylines into 256-dimensional vector per polyline using simple MLP. To generate the input to this MLP, concatenate the position
13

---

Agents
12222rg 1001211
2R2e2rg
RONTAFEE
Positional Embedding
MLP
Time Transformer
Agent Transformer
Roadgraph Transformer
Figure 5: An overview of the DJINN architecture. The main network structure is comprised of time, agent and roadgraph attention layers. Features are encoded using positional encodings and MLPs. The output of the network an estimate the de-noised agent states.
and padding mask for each polyline, along with any additional per-polyline features present in the dataset. For both datasets, the MLP is comprised four linear layers, with a hidden dimensionality of 256 and ReLU non-linearities.
Transformer Network
The main transformer backbone of DJINN is comprised of 15 transformer layers which perform self-attention over the time and agent dimension, and cross attention with the roadgraph encodings. We utilize the same transformer layers as those proposed in [30]. but modify the number of layers and their ordering. Specifically, include more time transformer layers as we found this produced smoother trajectories. All attention layers consume and produce 256 dimensional features per-agent state. We use four heads for each attention operation, and 1024 dimensional hidden state for the feed forward network. In the transformer layers, we use the pre-layernorm structure described in [49].
Due to batching and agents which are not tracked for the duration of the traffic scene, there is padding present in the agent feature tensor. The transformer layers account for padding in the scene by modifying the attention masks so that padded agent states are not attended to.
Output MLP
We use a two layer MLP with hidden dimension 256 to produce the final output for Fg. We produce three dimensional vector per agent state for INTERACITON and two-dimensional vector for Argoverse since headings are not provided in the dataset.
14

---

A.3 Training details
We train DJINN on two A00 GPUs for 150 epochs. We utilize the Adam optimizer with learning rate of 3E-4 and default values for 31 and 82. We use a linear learning rate ramp up, scaling from 3E-4 over 0.1 epochs. We set the batch size to 32. We clip gradients maximum norm of Training takes approximately 6 days to complete from scratch.
B Observation Distribution
We train DJINN over a variety of observation masks O by randomly drawing masks from a training distribution p(O). Table 5 outlines this training task distribution. We refer to the length of agent state history observation for each dataset as tobs and the total number timesteps as T. U4 indicates a uniform distribution over integers.
Table Task distribution for training DJINN. The training observation mask O sampled from this distribution with probabilities given in the rightmost column.
Task
Description
Probability
Predictive
Observe states where / [0, labs]-
50%
Goal-Conditioned
Observe states where E [0, [obe] and the final state of 3 random agents.
25%
Agent-Conditioned
Observe states where E [0, lobs] and the entire trajec- tory of 3 random agents.
10%
Ego-Conniitioee
Observe states where t [0, tobel and the entire ego- agent trajectory.
10%
Windowed
Observe states where i 10, (start] and E (!start + lobs, T] where lsttart U(0, lobs).
5%
Upsampling
Observe every toba/'T states, starting from Estart (0, tobs/
5%
Imputation
Randomly sample observing each state with probability tobs/
5%
C Additional Qualitative Results
Fig. 6 shows additional qualitative samples from DJINN on the INTERACTION dataset for a subset of the observation masks outlined in Fig. Each row in the figure corresponds a different tasks, abd each element of the row is a sampled traffic scene.
D Additional Quantitative Results
D.1 Effect of Observation Distribution
To enable test-time conditioning through classifier-free guidance outlined in section 6.2, we train DJINN on the observation distribution described in Appendix B. To quantify the effect of training on this distribution, we compare the sample quality of DJINN model trained trained on the full observation distribution to one which is trained exclusively on the "Predictive Task." Table 6 shows the impact of the observation distribution measured by trajectory forecasting metrics samples drawn from INTERACTION dataset scenes.
Table 6 demonstrates that training on the full mixture of observation masks somewhal reduces the predictive performance of DJINN when compared the model trained exclusively on the predictive task. However, the diversity of trajectories measured using MFD [40] increases when training on the more diverse distribution.
15

---

Predictive
BBB
Goal Conditioned
Agent Reactive
Up-Sampling
observed
generated
Figure 6: Additional generated traffic scenes from the INTERACTION validation set. Each row demonstrates samples generated using different observation mask.
Table 6: Comparison of trajectory forecasting performance for models trained with varying observa- tion distributions. Trajectory metrics are measured using samples per scene on the INTERACTION
validation set.
Observations 0.21
minADE
minFDE
Scene minADE
Scene minF DE
MFD
Predictive Mixture
0.49
0.35
0.91
2.33
0.26
0.63
0.45
1.17
3.11
D.2 Effect of Reduced Sampling Steps
The continuous time training procedure of DJINN enables test-time variation in the number of sampling steps. Table 7 outlines the effect reducing the number of sampling steps from the 50 steps which are used in all other experiments.
Table shows that reducing the number of sampling steps results in modest trajectory forecasting performance reductions up to 20 sampling steps across all metrics. Using 10 steps severely impacts the quality of sampled scenes across all metrics. As sampling time scales linearly with the number of sampling steps, reducing the number of sampling steps allows for performance runtime tradeoff.
16

---

Table Trajectory forecasting performance versus the number of timesteps used the diffusion sampling procedure. Trajectory f performance is measured using 6 samples per scene on the INTERACTION validation set.
Diffusion Steps
minADE
minFDE
Scene minADE
Scene minFDE
10
0.28
0.64
0.45
1.135
20
0.22
0.51
0.37
0.95
30
0.22
0.50
0.36
0,92
40
0.21
0.50
0.35
0.92
50
0.21
0.49
0.35
0.92
D.3
Model Runtime
We compare DJINN's runtime SceneTransformer [30], varying the input size as measured by the number of agents the scene. Runtimes are measured across 1000 samples on a GeForce RTX 2070 Mobile GPU.
Table 8: Average scenario generation time for DJINN and Scene Transformer across varying scene sizes.
Agent Count Scene Transformer
DJINN 50 Steps
DJINN 25 Steps
0.0126s
0.574s
I.15s
16
0.0140s
0.61
1.24s
32
0.017s
0.844s
1.69s
64
0.026s
1.40s
2.89s
17


Reviews:
[{"review_id": "FPnmGR6H3Y", "reviewer": "Reviewer_1Qe5", "summary": "DJINN (Diffusion-based Joint Interaction Network) is an innovative generative model that creates, edits, and forecasts multi-vehicle traffic scenarios in a stochastic manner. Leveraging diffusion models, DJINN also addresses the challenge of generating traffic scenes conditioned on a flexible configuration for the observation space. This flexibility of the model makes it well-suited for generating safety-critical events and out-of-distribution scenarios, which are extremely relevant for evaluating the performance of autonomous vehicles. Furthermore, by jointly diffusing the trajectories of all agents and conditioning them to customizable observation windows, the authors provide a fresh and innovative way of forecasting multi-vehicle traffic scenarios.\n\nThe authors validate the efficacy of DJINN by benchmarking against state-of-the-art trajectory forecasting methods using the popular Argoverse and INTERACTION datasets. The results show that DJINN outperforms existing models like Scene Transformer in joint motion forecasting metrics, demonstrating its promise for contributing significantly to the development and safety testing of autonomous vehicles. Moreover, DJINN’s flexibility is demonstrated through its successful generation of goal-directed samples, examples of cut-in driving behaviors, and editing replay logs.\n", "questions": "- How might DJINN be adapted to handle more complex scenarios, higher action spaces or other robotic domain?\n- Given that the inference time of DJINN it relatively slow, due to its diffusion structure and iterative estimation of the score function. Do you have any quantitative results on the tradeoff between computing time, performance and variation of network? (number of layer, depth, number of diffusion steps, etc)\n- How suitable would the model, as it is, be for a model of predictive control setting and real life deployment?", "limitations": "While the authors acknowledge some limitations of DJINN in the conclusion, particularly the relatively slow inference time, a more comprehensive and explicit discussion of these drawbacks would enhance the overall balance and depth of the paper.\n\nMoreover, this section could provide a more explicit roadmap for future research directions. While the paper showcases DJINN's potential, there is room to explore its applicability in other contexts, or the implications of further improving its inference speed. Articulating these directions can inspire and guide subsequent work in this line of research.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThe paper represents a highly original contribution to the field of autonomous vehicle simulation. The use of a diffusion model for generating joint traffic scenarios is innovative, and appears to be a novel application of this technique. The authors move beyond the conventional deterministic sets of trajectory forecasts, proposing a generative model to forecast joint future motion. DJINN's ability to draw traffic scenarios from a variety of conditional distributions further attests to its innovative design. The model also stands out in its ability to provide flexibility in terms of test-time diffusion guidance, offering a new perspective on traffic scenario simulation.\n\nQuality:\nThe overall quality of the research is high. The paper presents a sound background and solid related works section, creating a strong foundation for their argument. The authors have clearly demonstrated their understanding of the topic, using relevant and current research. Their methodology is rigorous, and the experiments performed on the Argoverse and INTERACTION datasets are thorough and well-executed. The results obtained are credible and validate the authors' claims about DJINN's performance and flexibility.\n\nClarity:\nThe paper is well-written and clear. It efficiently communicates the problems associated with the simulation of autonomous vehicle systems and convincingly presents DJINN as a potential solution. The structure of the paper is logical, and the flow of ideas is coherent, making it easy for readers to follow the authors' argument.\n\nSignificance:\nThe significance of this paper is recognizable. Simulating diverse, safety-critical and stochastic traffic scenarios is of paramount importance in the field of autonomous vehicles.  Its flexibility in controlling the conditioning of traffic scenes, coupled with its excellent performance on trajectory forecasting, makes it a valuable tool for researchers and developers in the field. Given the increasing interest in autonomous vehicles, this work is likely to have a high impact on both academia and industry.", "weaknesses": "While the paper represents a significant contribution to the field, there are a few areas where the authors could improve upon in the future iterations of their work:\n\n- Detailed Analysis of Limitations: While the conclusion does mention certain limitations, the paper would benefit from a more detailed and separate section dedicated to the limitations of the proposed method. A thorough examination of the constraints and failure cases of DJINN could offer readers a more balanced perspective and help to guide future research efforts.\n\n- Future Directions: The authors might also consider providing clearer guidance on the future direction of the work. While DJINN has shown promise in its current state, discussing potential extensions and enhancements would be valuable. For instance, how might DJINN be adapted to handle more complex scenarios, or different action spaces and robotic domain?\n\n- Qualitative Analysis of Failure Cases: Alongside quantitative performance metrics, a qualitative analysis of failure cases would provide more comprehensive insights into the model's performance. Discussing specific scenarios where DJINN fails to generate accurate traffic scenes would offer readers a better understanding of its potential weaknesses and areas for improvement.\n\n- More Sample Traffic Scenes: The paper could also benefit from a larger set of sample traffic scenes, either included in an appendix or made available online. These examples would provide a more tangible sense of the model's capabilities. Video demonstrations would be particularly effective, allowing readers to visually appreciate the flexibility and realism of the scenes generated by DJINN.\n\n- Ablation Studies: The paper could also greatly benefit from conducting ablation studies to understand the influence of different components or variations of DJINN's model architecture on its overall performance. A detailed ablation analysis can help to pinpoint which elements of the model contribute most to its successful generation of traffic scenarios. This could include investigations into the impact of the diffusion process, the role of different conditional distributions, or the effect of different types of state observations. Understanding these aspects in more detail would not only provide further insights into DJINN's performance and functioning, but also guide future optimization and refinement of the model.\n\nMinor comments:\nUnlinked reference at line 88.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank reviewer 1Qe5 for taking the time to consider our work, as well as for their overwhelmingly positive and constructive feedback. We consider the areas which the reviewer believes that our work can be further improved below:\n\n**Detailed Analysis of Limitations**: We agree that additional discussion of the limitations could improve the clarity and impact of our work. In order to address the reviewer's comment, we plan to include an additional ablation study in the final draft of the paper which we feel will provide the reader with a more intuitive understanding of our model and its limitations. Specifically we will look at how DJINN’s run-time scales under increased agents and scenario lengths. This is a well known shortcoming of transformer models, which by extension also affects DJINN itself. \n\n**Future Directions**: We note that a small discussion of future research areas is included in section 7 (line 321), however in the final draft we will further elaborate on these directions, including many of the useful suggestions provided by the reviewers.\n\n**Qualitative Analysis of Failure Cases**: We agree that consideration of failure cases can often provide insight into what the limitations of the model might be in practice. However during our experiments we found that failure cases were most often the result of poor dataset annotations, and not any fundamental underlying problem with our modeling approach. \n\n**More Sample Traffic Scenes**: We thank the reviewer for this suggestion, which we agree would enhance the quality of our work. We have attached a pdf containing an additional figure in our rebuttal showing additional qualitative results for our method. This figure provides samples using the predictive, goal conditioned, agent reactive, and up-sampling tasks outlined in Figure 1 of the main text. If accepted, we will add this figure to the supplementary. Additionally, in the final draft we can include a link to a paper web-page including video examples of our method.\n\n**Ablation Studies**: In order to address the reviewer’s concerns, we have included the following ablation result regarding the effect of varying number of diffusion steps on the quality of generated traffic scenarios. We evaluate our INTERACTION model using a variety of diffusion steps, evaluating metrics over a 6 trajectory samples per scene, and refrain from fitting a Gaussian mixture model.\n\t\n| Diffusion Steps | Ego minADE | Ego minFDE | Scene minADE | Scene minFDE |\n|----------------:|-----------:|-----------:|-------------:|-------------:|\n|              10 |       0.28 |       0.64 |         0.45 |        1.135 |\n|              20 |       0.22 |       0.51 |         0.37 |         0.95 |\n|              30 |       0.22 |       0.50 |         0.36 |         0.92 |\n|              40 |       0.21 |       0.50 |         0.35 |         0.92 |\n|              50 |   **0.21** |   **0.49** |     **0.35** |     **0.91** |\n\n\nReducing the number of diffusion steps decreases the predictive accuracy of the model, but improves the runtime, allowing the user to trade performance for speed. The other ablation which was requested in the questions section of the review regarding the varying of model layer depth will be included in the appendix of the final draft of the paper. \n\n**More complex scenarios**: We note that because DJINN is based on a transformer model, we expect that with larger datasets, the model's performance would continue to scale desirably. This property has been well studied in other transformer networks [Kaplan and Mcandlish, 2020]. While adapting DJINN to more complex robotics domains represents an interesting future direction, it is beyond the scope of this work. DJINN is tailored specifically to autonomous driving, and while the diffusion framework can be adapted to arbitrary state spaces, determining how best to learn in such state spaces would require additional consideration. We will add this as a suggestion for future work into our final manuscript. \n\nComputing time, performance and variation of network: As is discussed in Ablation Studies, we have included an experiment which considers the effects of varying the number of diffusion steps. In our response to reviewer 1Rcd, we have also measured the runtime of our method over varying numbers of agents and diffusion steps.  An experiment which varies the network structure will be included in the final draft of the paper.\n\n**Suitability in MPC**: DJINN could be adapted to a multi-agent, model predictive control setting (MPC), where a local dynamics model is known but the external agents’ models are unknown. MPC could proceed by taking an action, stepping the local dynamics model to determine the next egocentric state, then using DJINN to predict the states of all external agents in the scene. Such a method could function as a basic MPC algorithm, or could even be used for interactive simulation and driving policy evaluation. Additionally, we hypothesize that MPC might be useful in reducing collisions in generated scenarios, but evaluation of this hypothesis is beyond the scope of our current work.\n\nWe agree that investigation into how generative models like DJINN might be applied to real-world deployment represents an important research direction as autonomous vehicle technologies continue to evolve. However, considering the difficult research questions surrounding closing the sim2real gap and diffusion model runtime, DJINN would be unlikely to be suitable for onboard, real-time deployment. That said, we will add a discussion of these research questions to our final manuscript.\n"}, {"review_id": "eK1wt0ByQ0", "reviewer": "Reviewer_TiFk", "summary": "The paper proposes a diffusion model-based method for generating joint or conditioned predictions of traffic agents. The proposed method combines the EDM diffusion and SceneTransformer model architecture and provides different guidance for conditional predictions. The experiment results indicate comparable results of the proposed results with SOTA.", "questions": "It would be great if the author could give more analysis and explanations on why the performance of the proposed method doesn't show clear improvements. ", "limitations": "As mentioned in the paper, the long inference time limits the method to be more useful in practice.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is well-written and the proposed approach gives readers very useful information about how the two SOTA methods (EDM and SceneTransformer) could be combined and its potential performance.", "weaknesses": "In my understanding, the novelty of the paper is mostly on the combination of existing methods, which is OK but not outstanding.\n\nFrom the experiment results, the proposed method doesn't show clear improvements compared with other methods, which is a bit surprising given the combination of two powerful approaches. It would be great if the author could give more analysis and explanations on why this is the case. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to reviewer TiFK for their thoughtful review of our work and their praise of our paper’s clarity. Below, we have responded to the two areas of weakness which the reviewer has highlighted.\n\n**Novelty**: In regards to the novelty of our approach, we accept that our method combines the training and inference structure of EDM [21], while using a model backbone similar to Scene Transformer [30]. However we emphasize that in combining these two approaches we enable capabilities which are not present independently in either work. DJINN allows increased control over traffic scene generation through guidance and scenario editing. Specifically, sampling traffic scenes using arbitrary trajectory classes and editing scenarios are both capabilities which are not available in either work.\n\n**Performance**: We argue that the main contribution of our approach is not the predictive power of DJINN as a trajectory forecasting model, but the ability to sample traffic scenes from a wide variety of conditional distributions through guidance and editing techniques as outlined in section 5 and demonstrated in experiment sections 6.2-6.4.\n\nRegarding raw performance, we first highlight that DJINN is constructed to perform joint trajectory prediction and not marginal trajectory prediction. In joint trajectory forecasting, DJINN outperforms a comparable baseline as is shown in Table 2, section 6.1. \n\nWhen comparing our joint model against marginal methods in Table 1, we highlight that the standard trajectory forecasting metrics, minFDE and minADE, measure the minimum error over a small set of predicted trajectories. We argue that these metrics implicitly make a strong modeling assumption that the true distribution of driving behavior follows a finite component mixture distribution. \n\nThe non-generative trajectory forecasting baselines considered in our work incorporate this assumption by directly modeling discrete, deterministic trajectory sets. These sets contain the average driving behavior of each component in the mixture, but might not capture the true driving distribution. Alternatively, DJINN is trained to stochastically produce traffic scenarios which match the data distribution by minimizing a weighted variational bound with weaker assumptions about the structure of the underlying data distribution. \n\nWe argue that the slightly worse performance of DJINN relative to these models stems from the shared implicit assumption made by common evaluation metrics like minFDE and minADE and the baseline models we compare against. Utilizing the post-processing described in section 6.1 to approximate DJINN’s learned distribution with a six component Gaussian mixture results in reasonable forecasting performance under these metrics. However, we hypothesize that more direct minimization of these metrics through the shared mixture assumption described above is the primary reason for the performance discrepancy. \n"}, {"review_id": "0jblQTYYjI", "reviewer": "Reviewer_1Rcd", "summary": "The paper proposes a generative model for producing synthetic traffic scenarios. The proposed method uses a diffusion model and learns to predict multiple agent trajectories jointly in a scene. The main contribution to previous methods is the increased flexibility of conditioning which can use different observation masks (corresponding to different use-cases), goal/non-goal, and other vehicle properties. The method is evaluated on two publicly available datasets and shows competitive performance to its deterministic counterparts.", "questions": "See weaknesses.", "limitations": "The paper did not address the limitations and potential negative societal impact.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "DJINN seems like a natural step forward for the multi-agent trajectory prediction task. In theory, the generative framework of diffusion models should allow for better sampling of the data distribution and produce traffic scenarios with larger variability that previous methods such as Scene Transformer [30] and TrafficSim [44]. This is important because datasets with traffic scenarios are highly imbalanced, containing very few safety critical events which are usually more interesting. The paper illustrates this flexibility through several interesting qualitative examples of test-time conditioning. Furthermore, the paper is well written, providing clear motivation, problem formulation, and background to the reader.", "weaknesses": "Quantitative results in Tables 1,2 show comparable results to [30] in terms of forecasting performance but there is no quantitative experiment demonstrating the increased variability of generated traffic scenarios from the proposed approach. While this is shown qualitatively in sections 6.2, 6.3, and 6.4, and in my understanding [30] does not have the capability of test-time conditioning, I think a quantitative comparison (perhaps estimating variance over waypoint predictions) would still be useful. I would like to hear the authors' thoughts on this. \n\nThe authors mention that the choice of the reference frame for the agent states is important. Perhaps an ablation study would provide more insight.\n\nWhat is the effect of using the observation masks during training besides defining different use-cases? How would the capability of the model to produce traffic scenarios with large variability be affected if only the \"Predictive\" mask was used?\n\nPlease include a runtime comparison to baseline methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to consider our work, and for providing positive, and constructive feedback. We address their concerns below:\n\n**Demonstration of increased variability**: We agree that such a comparison should be considered in the evaluation of our model in comparison to the baseline. One possible metric we could consider is the maximum final discrepancy (MFD) [38], which we have used in the “Effect of observation mask” subsection of this rebuttal to measure the variation of generated trajectories. If the reviewer believes that the inclusion of this evaluation would improve the clarity and impact of our work, it can be included in the final version of the paper.\n\n**Test-time conditioning**: We agree with the reviewer that surfacing rare events such as safety critical incidents is challenging with imbalanced driving data. The test time conditioning introduced in DJINN is one answer to this problem. Specifically, using classifiers of rare driving behavior and editing scenarios are two approaches introduced which allow sampling from conditional distributions of interest which are not available in prior methods.\n\n**Reference Frame**: Although the choice of reference frame is a key design choice, it is not easily modified without significant changes to the model architecture. This is because DJINN is specifically designed for a joint global frame, making extensions to an egocentric frame difficult . Therefore, we argue that such ego-centric prediction is outside the scope of this work, though we agree that it is a promising avenue for future research.\n\n**Effect of observation masks**: As the reviewer mentions, the primary reason for training over different observation masks is to define different use-cases at test time. To measure the impact of the observation mask distribution, we have compared the diversity of samples from a model trained with the “predictive” mask to those produced by a model trained on the full observation mask distribution on the INTERACTION validation set. We utilize the maximum final discrepancy as our diversity metric [38] which measures the maximum distance between all pairs of trajectories in a trajectory set.\n\n| Model                \t| Mean MFD |\n|--------------------------|---------:|\n| Predictive Only      \t| \t2.33 |\n| Observation Distribution |3.11 |\n\nFrom our experiment, we note that training over the observation mask distribution increases the diversity of trajectories produced by DJINN. However, we reiterate that the purpose of the observation distribution is primarily to enable test time guidance using classifier-free guidance and conditional generation of trajectories using arbitrary observation masks.\n\n**Runtime**: We have attached a comparison of the runtime of DJINN with [30] which will be added to the final supplementary. We note that the runtime can be lowered at the expense of performance as demonstrated in the rebuttal to reviewer 1Qe5, and that distillation is a promising avenue of future work to lower the computational cost of diffusion models.\n\n| Number of Agents | Scene Transformer | DJINN - 25 Steps | DJINN - 50 Steps |\n|-----------------:|------------------:|-----------------:|-----------------:|\n|            \t8 |       \t0.0126s |       \t0.574s |       \t1.149s |\n|           \t16 |        \t0.014s |       \t0.611s |       \t1.238s |\n|           \t32 |        \t0.017s |       \t0.844s |       \t1.693s |\n|           \t64 |        \t0.026s |       \t1.404s |       \t2.891s |\n"}, {"review_id": "ui1WDlV6iS", "reviewer": "Reviewer_wJNr", "summary": "The paper deals with the problem of generating vehicle trajectories at the scene level conditioned on a map and some known observations (e.g. past or future states). They propose a diffusion model based on SceneTransformer that is trained with observation conditionings with random masking to reflect multiple desired downstream applications including prediction, goal conditioning, and imputation.  Experiments show reasonable results on trajectory forecasting on both Argoverse and INTERACTION datasets, and demonstrate the flexibility of the model to several tasks including generating scenes based on high-level concepts (cut-in) and scenario editing.", "questions": "Overall, the paper is a nice exploration and demonstration of how diffusion can be applied to traffic scenario modeling, and I tend to think it would be a good contribution to the community. But in the rebuttal, it would be really good to see more qualitative results and for the authors to clear up some of my confusion on the masking and classifier-free guidance procedures.\n\nI had a few questions in Sec 4 that didn’t weight heavily in my rating:\n* Eqn 5: why is the loss on $x$ and not $x_0$? Is the entire trajectory supervised (observed + generated) or just the non-observed parts?\n* L186: why are the agent states called “latent”? Are they already embeddings of some kind?\n* It’s interesting that deterministic sampling is better quality than stochastic since this is not the case in other domains – any intuition here?\n\n============== After Rebuttal ===================\n\nAfter the rebuttal and discussion, I have decided to raise my score to Accept. In the rebuttal, the authors addressed most of my concerns, clarified various technical points, and added more qualitative results that demonstrate the variety of applications.", "limitations": "Limitations are discussed in Sec 7. It might be good to mention some limitations/failures specific to DJINN and not diffusion in general, e.g., mention that the model is not always realistically consistent with conditioning observations?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "While no technical component of DJINN is particularly novel, the paper presents a nice comprehensive exploration of using diffusion for traffic scenario modeling. This includes all different types of masking/conditioning, classifier-free guidance, classifier guidance, and scene editing with SDEdit.\n\nSec 3.2 is a succinct, yet informative, introduction to diffusion which will be good even for the unfamiliar reader in the driving community. In general, the paper is relatively self-contained, well-written, and easy to follow.\n\nThe conditioning approach makes the model very flexible to many different kinds of observations that enable use in several important downstream applications.\n\nThe paper goes beyond showing the usual motion forecasting applications (although it shows reasonable results on this too). The use of classifier guidance in Sec 6.3 to allow creating scenarios with more abstract specifications is a cool proof-of-concept that could be really useful for AV testing. Similarly for the scene editing application in Sec 6.4.", "weaknesses": "In some cases, it looks like conditioning the model on input observations is not strong enough to be consistent with the observations. E.g. Fig 1 bottom row for Goal Conditioning, Up-Sampling, and Imputation the generated parts of trajectories in orange do not always align well with observations in blue, i.e. there are jumps or discontinuities. It may be necessary to use test-time guidance (as in CTG [52] and [Trace and Pace, Rempe et al., CVPR 2023]) to better enforce consistency. Also, it would be good to have an evaluation to quantify this inconsistency. E.g., for up-sampling and imputation the generated portions could be compared to ground truth, or for goal conditioning how close the vehicle gets to the goal. \n\nSome details of the masking procedure to allow this conditioning were also not clear. Does the model only output predictions for timesteps which there were no observations? Or does it generate a full trajectory and only the unmasked portion is visualized in the figures? If the latter, is the masked portion (i.e. the part of the trajectory given as input) supervised during training?\n\nThe classifier-free guidance (CFG) formulation was also a bit confusing. The difference between $x_{obs}$ and $x_{cond}$ in Sec 5.1 was not apparent until later in Sec 6.2. Usually CFG operates by dropping the entire conditioning from the model in the right-hand term of Eqn 6, so I’m wondering why only some of the conditioning (e.g. the goal state in Sec 6.2) is arbitrarily dropped in the proposed formulation? In this example (and in others like motion forecasting), it seems the past conditioning should also be dropped. I also think on L275-276 it may be more accurate to say that the CFG controls the emphasis of the conditioning to the model instead of controlling the “spread of agents’ trajectories”: the reason the spread reduces may be because the goal position conditioning is emphasized so more samples are guided closer to this goal.\n\nThe paper and supplement currently have only a few qualitative results, making it hard to comprehensively evaluate trajectory quality. Given the range of applications that the masking enables, it would be great to show several examples from each of these applications. Videos would be ideal, but at least trajectories should be colored with a gradient over time or something rather than a solid color over the whole trajectory. \n\nThe observation distribution is detailed in Appendix B, but I’m curious about how this distribution was determined and how much it affects the final capabilities of the model? It seems balancing these probabilities would be very important to ensure good results, so I’m wondering does the model do significantly worse when trained on all tasks jointly than trained on one individually?\n\nOne of the most useful applications of the proposed method could be in simulating scenarios for AV testing, but the current generated trajectories are quite short (at most 5 sec) and limits the kind of scenarios. I’m interested to see if the model can handle longer generated scenarios (~10 sec), and if the trajectories maintain realism using metrics like off-road and collision rate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Consistency with input observations: We agree that in some examples, DJINN produces samples which are inconsistent across time. We also agree that these inconsistencies may be reduced by utilizing test time guidance. Specifically, classifier-free guidance as outlined in section 5.1 and demonstrated in section 6.2 might be used to improve the quality of these samples. In addition, iterative classifier guidance proposed by CTG is directly compatible with the classifier guidance outlined in section 5.2 and is a promising avenue for future work. We propose including a supplementary figure showing the effect of classifier-free guidance over a variety of guidance weights on the samples from Figure 1 in the final draft. While “Trace and Pace” [Rempe et al., CVPR 2023] is concurrent work by the NeurIPS standards and focuses on pedestrians instead of vehicle motion modeling, we believe it adds additional context for the reader and therefore will add it to our related work section.\n\n**Masking Procedure**: DJINN only produces predictions for agent states which are not observed. In Figures 1 & 2, we visualize the generated, unobserved states in orange and the observed states, which are conditional input to the model, in blue.\n\n**Classifier Free Guidance Formalism**: Thank you for identifying that the distinction between $x_{obs}$ and $x_{cond}$ in section 5.1 is unclear. We will modify the introduction of $x_{cond}$ to include a statement indicating that $x_{cond}$ could be a future goal state to improve the clarity.\n\n**Additional Conditional inputs in CFG**: Regarding your question about our classifier free guidance formulation, we agree that in many applications of classifier-free guidance (for example class conditioned image generation) a completely unconditional score estimate is combined with the conditional score estimate. From [13]:\n\n$ \\nabla \\log p(\\mathbf{x}_t | y) =  (1-w) \\nabla \\log p(\\mathbf{x}_t) - w \\nabla \\log p(\\mathbf{x}_t | y)$\n\nWhere x is the diffusion variable, y are conditional inputs and w is the guidance weight. However, it is also valid to add additional conditioning to every term:\n\n$ \\nabla \\log p(\\mathbf{x}_t | y, z) =  (1-w) \\nabla \\log p(\\mathbf{x}_t | z) - w \\nabla \\log p(\\mathbf{x}_t | y, z)$\n\nIn section 5.1, we define $y$ as $\\mathbf{x}_{cond}$. Here, $z$ is all the other conditional information provided to the model, which in our application includes the map as well as other observed agent states. Our removal of specific agent states during guidance is therefore not arbitrary but instead an intentional design choice which follows the conventions of classifier-free guidance with added conditional information.\n\n**Phrasing of guidance strength**: We agree with the reviewer’s suggestion about the phrasing of line 275-276 and will update the final manuscript with this modification.\n\n**Additional Qualitative Results**: We have supplied some additional composite images showing scenario samples using a variety of observation masks. We agree that gradients on the qualitative result figures might improve clarity, and will update our figures with this modification.\n\n**Impact of the Observation Distribution**: The mixture weights for the various tasks in the observation distribution were chosen empirically. However, we have included an additional experiment in our rebuttal investigating this choice. \n\n| Model                \t| Ego minADE | Ego minFDE | Scene minADE | Scene minFDE | Mean MFD |\n|--------------------------|-----------:|-----------:|-------------:|-------------:|---------:|\n| Predictive Only      \t|   \t0.21 |   \t0.49 |     \t0.35 |     \t0.91 | \t2.33 |\n| Observation Distribution |   \t0.26 |   \t0.63 |     \t0.45 |     \t1.17 | \t3.11 |\n\nThe table above shows the difference in performance on the INTERACTION dataset between a model trained on only the predictive task and one trained on the full observation mask distribution as measured using  6 trajectory random samples. As expected, we find the model which is optimized solely on the predictive task performs better on that task than the model which is trained on a wider distribution of tasks. We expect that scaling the network size of DJINN would likely reduce this difference. \n\n**Scaling**: We have only trained our model on datasets with scenarios of up to 5 seconds in length, as this is what is commonly used in challenges related to these datasets. We agree that generating longer scenarios is an important avenue of future work, but it is beyond the scope of this work. Therefore, in the final draft of our paper, we will discuss increasing the length of generated scenarios as an avenue of future research.\n\n**Eqn 5**: The loss we optimize is squared L2 error between the output of D_\\theta and the noiseless unobserved states in the scene. We agree that the notation of equation 5 could be improved. In the final version of our paper, we will clarify the notation used in this equation.\n\n**L186**: We use the term “latent” as a synonym for “unobserved”, following the graphical model convention in which variables can be either latent or observed. We will update the text to improve clarity surrounding this point.\n\n**Deterministic vs Stochastic Sampling**: We note that EDM [21] also found that deterministic sampling performed better than stochastic sampling in their experiments. Since we utilize their diffusion parameterization, we are not surprised that deterministic sampling performs better in our experiments also.\n"}]

Author rebuttal:
We would like to thank all of the reviewers for their time in considering our work and their thoughtful comments which we believe will help to improve the quality of our submission.

In response to requests for additional qualitative results, we have provided an attached pdf which contains a composite image of multiple traffic scenes generated by DJINN under a variety of observation masks.

Meta-review: The paper proposes a diffusion model for generating joint traffic scenarios. The reviewers appreciate the clear motivation, good presentation and comprehensive ablations despide of a limited novelty on components of the method. All the reviewers agreed to accept the paper after the rebuttal. Therefore, the AC recommends accepting the paper.
Decision: Accept (poster)