SceneStreamer: Continuous Scenario Generation as Next Token Group Prediction
Authors: Zhenghao Peng
Yuxin Liu
Bolei Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Autonomous driving
Closed-loop simulation
Scenario generation
Primary area: applications to robotics, autonomy, planning
TL;DR: SceneStreamer models scenario generation as next token group prediction, enabling dynamic and realistic traffic simulation with a unified autoregressive model. It supports agent insertion, motion rollout, and improves RL policy robustness.

Abstract: Realistic and interactive traffic simulation is essential for training and evaluating autonomous driving systems. However, most existing data-driven simulation methods rely on static initialization or log-replay data, limiting their ability to model dynamic, long-horizon scenarios with evolving agent populations. 
We propose SceneStreamer, a unified autoregressive framework for continuous scenario generation that represents the entire scene as a sequence of tokens, including traffic light signals, agent states, and motion vectors, and generates them step by step with a transformer model. This design enables SceneStreamer to continuously introduce and retire agents over an unbounded horizon, supporting realistic long-duration simulation. Experiments demonstrate that SceneStreamer produces realistic, diverse, and adaptive traffic behaviors. Furthermore, reinforcement learning policies trained in SceneStreamer-generated scenarios achieve superior robustness and generalization, validating its utility as a high-fidelity simulation environment for autonomous driving. More information is available at https://vail-ucla.github.io/scenestreamer/ .

Published as a conference paper at ICLR 2026
SCENESTREAMER: CONTINUOUS SCENARIO GENER- ATION AS NEXT TOKEN GROUP PREDICTION
Zhenghao "Mark" Peng, Yuxin Liu, Bolei Zhou University of California, Los Angeles
ABSTRACT
Realistic and interactive traffic simulation is essential for training and evaluat- ing autonomous driving systems. However, most existing data-driven simula- tion methods rely static initialization or log-replay data, limiting their abil- ity to model dynamic, long-horizon scenarios with evolving agent populations. We propose SceneStreamer, a unified autoregressive framework for continuous scenario generation that represents the entire scene as sequence of tokens, in- cluding traffic light signals, agent states, and motion vectors, and generates them step by step with a transformer model. This design enables SceneStreamer to continuously introduce and retire agents over an unbounded horizon, supporting realistic long-duration simulation. Experiments demonstrate that SceneStreamer produces realistic, diverse, and adaptive traffic behaviors. Furthermore, reinforce- ment learning policies trained in SeeeneSttramm--geeeeee perior robustness and generalization, validating its utility as a high-fidelity sim- ulation environment for autonomous driving. More information is available at hattps://vail-a httpaaaeeecee.iiiioo.oooo/eeeeee
scenarios achieve su-
INTRODUCTION
Simulating realistic and diverse traffic scenarios is vital for the development and evaluation of au- tonomous driving systems. Simulation enables safe, cost-effective, and repeatable testing of driving policies without relying on real-world deployment. However, most existing frameworks use static traffic generation methods, such as replaying logged trajectories from real-world datasets (Li al.) 2023 Dosovitskly et al. 2017 Gulino et al.) 2023). Although faithful to real driving behaviors, they lack interactivity as background agents do not respond the ego vehicle's actions, limiting their utility for closed-loop evaluation.
Recently, data-driven generative models have emerged to learn to synthesize traffic scenarios from real data, offering path toward richer and more realistic simulations (Zhang et al. 2025a Rowe cct al. 2025). Learning-base traffic simulation is commonly framed as a motion prediction problem: given a history of agent states in scene, including map, signals, and initial state of agents, policy generates the future trajectories of all agents. However, most such models are trained as a one-shot prediction model (Ngiam et al.) Shi et al. 2022 Pronovost et al. 2023) and do not explicitly model interactions between agents during the prediction horizon, which leads to covariate shift when they unrolled in the simulation. Small prediction errors can compound, causing the simulator to visit out-of-distribution states and produce unrealistic outcomes. Recently, autoregressive models have been proposed to better fit into the driving behavior modeling, especially the context of closed-loop simulation (Suo et al.) 202T] Zhang et al.) [023b] Seif et al.) [023] Kamenev et al.) 2022]. However, these models still rely on the provided initial states of agents and miss the diversity that emerges from the initial layout of traffic participants. Some other works propose generating the initial conditions and then conducting motion prediction based on these conditions (Feng et al.) 2023 Bergamini et al. 2021 Tan et al.) 2021). This separation can be inefficient and inflexible, as it prevents the model from sharing context between the initialization and motion prediction phases. traffic participants to enter the scene over time. But in reality, traffic is an open system: new participants continuously enter the scene while old ones leave (e.g., vehicles turning into from side street).
It also means the number of agents is fixed at initialization, disallowing

---

Published as a conference paper at ICLR 2026
Agent State
Light
Agont Motion S.
2ND-1
Mepit
Motion Prediction
Scanario Generation
SceneStreamer
Real-world Map
Scenario Densification
Closed Hlop Simuiation
Figure 1: SceneStreamer enables unified scenario generation via autoregressive token predic- tion. We represent a dynamic driving scene using structured sequence of discrete tokens grouped into traffic light, agent state, and agent motion tokens. SceneStreamer generates these tokens step- by-step on top of static map tokens, allowing flexible and fine- fine-grained simulation. Our unified model supports diverse downstream applications: motion prediction, full-scenario generation from scratch, scenario densification by injecting new agents, and closed-loop simulation for training self-driving planners.
To address these gaps and enable continuous scenario generation, we propose SceneStreamed" generative traffic simulation framework that models the entire scenario, including agent states and motions, as single sequence of tokens. SceneStreamer uses a unified autoregressive model to generate both the agent state and agent motion at every step. We tokenize different categories of agents, such vehicles, pedestrians, and cyclists, in the same way, with different category embed- dings added to their tokens. Notably, SceneStreamer is flexible and can adapt to different tasks, including motion prediction, state initialization, scenario generation, and scene editing (e.g., adding new agents and densification). by choosing different tokens to be state-forced, while others are sampled. We implement carefully designed agent state tokenization pipeline that the model can effectively handle heterogeneous agent types and map context when adding new agents. Fi- nally, demonstrate that using SceneStreamer generate training scenarios leads significant pannforeement-leannnn-----ee
improvements in downstream planner performance. on ScemeStreamer geeeerrreeeeeee scenarios exhibit greater robustness and better generalization to novel environments. We summarize our contributions below:
planners trained
SceneStreamer employs a single autoregressive model that produces both agents' initial states and their motion trajectories as part of one continuous token sequence over long horizons. This unified approach ensures consistent conditioning between where agent starts and how it moves, addressing the inflexibility of prior two-stage models.
1) Unified State & Trajectory Tokenization:
2) Agent State Autoregressive Generation: We design novel generation scheme for agent states by autoregressively rolling out the agent state tokens and generating the map-based relative states of agents, i.e. the agent's 's type, its map location, and its detailed kinematic state. This allows the model to accurately place agents specific map segments (e.g., lanes) and generate realistic state details (position, heading, velocity, etc.) in a compact, learnable representation.
3) Versatile Capabilities: By dynamically state-forcing different token groups, SceneStreamer is versatile and applicable to various tasks, including motion prediction, traffic simulation, sce- nario generation, and scene editing. We demonstrate that training autonomous-drívin planners in www.tttramee-geeerarreeee scenarios yields more robust and generalizable policies, indicating that SceneStreamer can serve as both a scenario generator and an effective data augmentation tool for closed-loop simulation.
'An earlier version of this work used the name "InfGen". To avoid confusion with the concurrent work (Yang et al. 2025), we refer
method SceneStreamer throughout this paper.
In this paper. we use the term "state-force" to describe directly feeding reconstructed tokens (e.g., agent states at the current step) hack into the model, bypassing the generative process. This is distinct from the conventional "teacher forcing" used in sequence modeling, where ground-truth tokens are fed.
2

---

Published as a conference paper at ICLR 2026
2 METHOD
Scenario Generation. A driving scenario comprises (1) static map context M (vectorized lane segments, crosswalks, etc.) and (2) dynamic entities including traffic-lights (1
Ne and traffic agents {ar Ji=l that evolve with time t. Here NTL denotes the number of traf- fic lights and N₂ denotes the number of agents at step Each traffic-light state 1(4) (I, 8) contains 2-D position and discrete signal
E yeleen, yellow, red, unknown] while each agent state a.") (xr.3. y. Uzz Vy: 1, w, h) encodes pose, velocity, heading. category c e (vehicle, pedestrian, cyclist) and length, width and height of the 3D bounding box of the agent. Compared to conventional motion prediction task, which assumes fixed agent set I (1, NN.
and access to agent history (aiz) and forecasts future trajectories (8441:Thee scenario gen- eration must create the initial agent set and continually inject new agents, traffic-light changes, and motions over a horizon T: (s...... ....,rrrr () with S. = (0( (a. ().
2.1 SCENARIO AS A TOKEN SEQUENCE
We cast scenario generation as a next-token prediction task: map tokens <MAP> followed each step by traffic-light tokens <TL>, agent-state tokens <AS>, and agent-motion tokens <MO> and form a single autoregressive token sequence: X1:7 <<MP>;;<TL>>> <AS>, <MO>);;<<LL>> <AS>, <MO>)2; Given all tokens X<t gener- ated so far, the model predicts the next token next set of tokens (a(x, xxx) and samples øt- Following this idea, we develop SceneStreamer, a unified transformer that sees the whole history and rolls out the scenario step-by-step, enabling fine-grained, closed-loop generation and smoother downstream simulation integration.
Map Tokens <MAP>. A map segment (e.g., road line, stop sign, crosswalk) is represented as a polyline of up to N₂ ordered 2D points with semantic attributes. We denote all M segments as Smap E TR. Rx Mx,xCC where C' is the per-point feature dimension (e.g., position, road type). A PointNet-like encoder (Q et al. 2017) yields features {m; which are passed through the Scen- eStreamer Encoder produce the map features: m' SceneStreamerEEn(()) To support cross- attention in the decoder, we assign each map segment a unique discrete index i (its map ID), and embed it into the map token:
<MAP>; m [[] + (mbMaplID(i) @ gi
M.
where EmbMaplD is learned embedding table. @ denotes we will record the geometric information gi of map segment i, which includes its center position and heading, and use it to participate the relative attention. We defer the discussion of relative attention to Sec. 2.3 The map ID will be used to refer a map segment during agent state generation (Sec. 2.2 p. These map tokens { {<MAP>> >>} provided by the SceneStreamer Encoder and are kept fixed during simulation, serving static cross-attention keys/values for all decoder layers.
Traffic Light Tokens <TL>. Each traffic light is represented by single token per step. We encode the traffic light's 's discrete state (green, yellow, red, or unknown), its identifier, and the ID Ag of the map segment it resides in and construct the traffic light token for light k at step
<TL>k, EmbState(s*..) + EmbTLID(k) EmbMapID(Aa) & gk: k 1,..,, NTL. (2)
where s E {G. U} is the signal state, A is the discrete map segment ID the light is attached to, and ge is its cemporal-geometiee context (position, orientation and current timestep). As with map tokens, & indicates that ga participates in relative attention (Sec.) 2.3).
Agent State Tokens <AS>. For every active agent, including newly injected agents at step t, SceneStreamer uses set of four agent state tokens that collectively encode the agent's dynamic and semantic state. These states include positions, headings, velocities, shapes and agent cate- gories. As shown in Fig. A),, each agent present at step i is represented by four ordered tokens: (<SOA>>, <TYPE>,, <MS>; >, <RS>x)> Here <SOA> is the of-re-t- flag, <TYPE> the categorical token in vehicle,, pedestrian, eyclist), <MS> is the index of the map segment where the agent resides as shown in Fig. B). <RS> is the relative states of agent w.r.t. the selected map segment. We defer the detailed composition of each token to the Appendix.
3

---

Published as a conference paper at ICLR 2026
(N) Theee Group Stuckin
Tokera brt
Tokaro at hr2
Group Alerrton
COOCOOOOO WWW.OOOOOOOOOOOOO
Takers #E51
Tokerra
SceneStreamer
Maps
Taearo 41
COOO
Malie
Servet
Tiu
HCAl
A5
Tikere t-0
Takeno a7
Map Tokena
Trant a Light Tooars
Agert Store Takono
Moxion Tahons
Figure 2: autoregressively generates a sequence of tokens representing a full traffic scenario. Each simulation step consists of traffic light tokens (purple), agent state tokens (blue), and motion tokens (green), conditioned on static map tokens (red). This structured tokenization enables step-wise wise rollout of the dynamic scene and allows new agents to be introduced at any timestep. (B) Grouped causal attention governs how tokens interact: each token attends densely within its group and logically preceding groups, while also incorporating cross-timestep context (e.g., agents attend to their own history). This attention design encodes semantic causality (e.g.. agent motion depends on agent state, which depends on map), enabling fine-grained closed-loop simulation with coherent agent behaviors.
The tokenization and attention mechanism SceneStreamer. (A) SceneStreamer
The most interesting token is the relative state token. Specifically, relative state ri is a 8D vector, each dimension representing (1, w. h, u, U, deb, Ugg U2), where (1, w, h) is the agent's physical dimensions (length, width, height), (u, v) is the longitudinal and lateral offset from the centerline of map segment Ae dy the heading residual relative to the map segment's s orientation, (va, Ug) is the velocity vector whose direction is in the frame of the map segment. The relative states r; can be autoregressively generated by the relative state head as shown in Fig. C) (see Sec. [.2]. Representing agent states relative to local map segments allows for a unified and compact token vocabulary, avoiding the need to discretize the entire map globally and enabling scalable scenario modeling.
field in the agent's relative state vector: ry
Motion Tokens <MO>. To model agent motion, SceneStreamer predicts a motion label for each agent, parameterized as a pair of acceleration and yaw rate (a,w). Given an agent's is current state, the next state predicted using a first-order bicycle model. We bucketize the acceleration and yaw rate space into uniform bins. To obtain the ground-truth (GT) motion token, enumerate all candidate motion labels (a, w) combinations and search for the best candidate with least Average Corner Error (ACE), the mean error between the 2D bounding boxes of candidate and GT. This strategy ensures that both position and heading are tightly aligned with GT and mitigate the compounding error in the tokenization of GT trajectory.
For an agent at a timestep 6, we use a motion token <MO>> to encode the motion label and the identity-related context. The motion token is computed as:
<MO>,, EmbbMtioo((J..a) + EmbType(c;) + EmbAID(1) + Emb EmbVel(v.) + EmbShape(s:) g:
(3)
where Pi,r is the motion label, q is the type of agent, is agent's ID, v, is a 2D vector representing the agent velocity in local frame, and s; is 3D vector of agent's shape (length, width, height), gi is a 4D vector encoding the temporal-geomette information (agent's s current global position, heading and time step). Note that the embedding tables EmbType and EmbAID are shared with <AS>.
2.2 AUTOREGRESSIVE SCENARIO GENERATION
SceneStreamer an encoder-decoder model. The SceneStreamer Encoder processes the in- formation of map segments and output {<MAP>>}. The SceneStreamer Decoder, denoted by SceneStreamerDec, autoregressively generates tokens in a step-by-step manner. As demonstrated in Fig. A). in each step, SceneStreamer first generates set of traffic light tokens <TL> predict- ing the next state of traffic light signals, then it generates the agent state tokens <AS> one-by-one. Finally, the motion tokens <MO> of all agents are generated.
Traffic Light Tokens <TL>, All traffic light tokens are generated in a single batch at each step, enabling the self-attention between nearby traffic lights. The output is obtained from the traffic light
4

---

Published as a conference paper at ICLR 2026
(A) Agent Tovarra
(8) Map Token
Mar
(C) Falutive Stateo
where
Bupies balature
Saadiinss exyraming 104)
Rebatie Shdee
conainn:
Save
50000032
228 30
Head
Typa
Map
Grasel
SceusSteanee
Map
SceneStreamer
Auti Mugssiiiyy prodec agen cAde sivve to The mas region
Ages TiPS
Mopii!
Rebadt
Figure 3: The design of agent state generation. (A) Each agent's state is encoded as 4 tokens. We first predict the agent type, select a map ID) where the agent resides, then predict the relative states. (B) Before obtaining the agent state, we first select a map segment the "anchor" where the agent should reside. (C) Feeding in the Map ID, we use the output token as the condition and call the Relative State Head, which a tiny transformer, to autoregressively generate the relative agent states, including shape, position, heading and velocity.
head, a MLP layer (-adTL(-) mapping the decoder output to the probabilities of four discrete states: {green, yellow, zed, unknowa):
(sK,ch ~ HeedTT
E NR-
(4)
Agent State Tokens <AS>. As shown in Fig. A),, for one agent, there are four tokens used to represent the agent state. In the test time, we will first sample agent type from the distribution produced by the agent type head: c Then, as shown in Fig. B), the model will select one of the map segment Ay: X HeadMaplD(sceeeStremmete <<TYE>))) After selecting a map segment X; and generating the associated map segment <MS> token, we condition on the decoder output of <MS> to generate the agent's full kinematic and shape attributes. As illustrated in Fig. CC. a dedicated module called the Relative State Head, a small Transformer decoder with AdaLN (Perez et al. 2018)) is used to au- toregressively generate a sequence of 8 tokens, each representing field in the relative state vector, conditioned by the latent vector from the decoder:
(5) Here, (1, we h) is the agent's length, width, height, (u, v) is the longitudinal and lateral offset from the center of map segment A₁ de is the relative heading to the map segment's orientation, (Ua, v2y) is the relative velocity vector. <SOS> is the -f-quunneeeeeeee indicator. We bucketize the continuous features L, w. h, u, otb Un, vy so the transformer can predict categorical distributions on them.
(1, w, h, u, u, 5v, Uxy Uty) ~ HeadRS (SceneStreamerDee HeeuBBrccos--SSnSSteeeneeDDe <<MS>)).
For existing agents that persist from the previous timestep, SceneStreamer bypasses the relative state prediction head and instead deterministically state-forces their agent state token using the map segment and relative states. Here, state-forcing denotes replacing predicted tokens with recon- structed state tokens whenever the agent's 's current state is already known. Note that state-forcing will not introduce information leak in inference as we don't read ground-truth agent state from data. This allows SceneStreamer to seamlessly unify dynamic agent injection (via sampling) and agent motion continuation (via foreino ensuring closed-loop autoregressive simulation across variable-length agent sets. Unlike prior methods such TrafficGen (Feng et al.) 2023), which gen- erate all agent state attributes simultaneously in a flat and unstructured output head, SceneStreamer decomposes the generation into causally constrained sequence and thus can better ensure semantic and physical consistency.
Motion Tokens <MO>. The motion head predicts each agent's motion label single categorical token from a 2D discretized space of acceleration and yaw rate. Specifically, we define a flat vocab- ulary, where each token corresponds to a unique pair (a, drawn from uniformly quantized grids A and Ω. A motion prediction head is used to obtain the probability distribution over motion labels. At inference time, we apply top-p (nucleus) sampling to select the motion labels while all motion labels at step are generated in a single batch:
(p..) ~ HeadMotion (eenaattiooSSSeeSSSeemmeeeeeeee
(6)
5

---

Published as a conference paper at ICLR 2026
Motion tokens of all agents generated in one batch as the traffic light tokens, enabling attention between neighboring agents. The sampled token Pi, is then mapped back to its corresponding (a,w) pair, and passed through first-order kinematic update rule to compute the next state (see Appendix). At an agent's first appearance, a special label Mstnnt is used to get <MO>. For continuing agents, the input motion token is simply the previously predicted token 14,
Each prediction head operates only on its associated tokens. This modular structure allows Scen- eStreamer handle heterogeneous outputs while maintaining unified sequence modeling.
2.3 MODEL DETAILS
Token Group Attention. We design token group attention mechanism, ensuring the causality while allowing effective information communication. As shown in Fig. BB) the rules are (1) tokens within the same group can attend each other freely (e.g.. motion tokens attend to other motion tokens at the same step); (2) tokens belonging to the same object (agent or traffic light) in a later step can attend the tokens of the same object earlier; and (3) every group of tokens can attend to the existing contexts at current or last step. For example, <MO>> can attend to current <TI>> <TL> can attend <MO> at last step, etc.
Relative Attention. We use relative attention biases between tokens, computed from (Ax, Ay, Au', At), modulate attention weights, following previous work on query-centric tention (Zhou et al. 2023 Shi et al. 2023 Wu et al.) [004). This makes the input token sequences information of the object, which eases model's training. A KNN mask restricts attention to spatial neighbors for scalability.
unaware the global
Model Architecture. SceneStreamer adopts an encoder-decoder architecture. The encoder embeds information of all map segments to static map tokens, which are cross-attended by the dynamic tokens in the decoder. The decoder generates heterogeneous output via different prediction heads as we discussed in Sec. 2.2 Each decoder layer combines I) cross-attention between dynamic tokens and static map tokens with 2) self-attention over the dynamic tokens using a structured group-causal mask Fig. B), enforcing semantic and temporal dependencies across token types. As all prediction heads output categorical distributions, SceneStreamer can be trained end-to-end using cross-entrue loss.
EXPERIMENTS
We evaluate SceneStreamer on suite of tasks assess the quality of its generated scenarios and its utility for downstream applications, particularly reinforcement learning (RL) planner training. Our experiments aim to answer the following questions:
Does SceneStreamer generate realistic and diverse agent states comparable real-world logged data?
Can SceneStreamer serve versatile simulation platform for motion prediction and scene generation?
Does training an RL planner in SceneStreamer--eeerrate scenarios lead to improved perfor- mance and robustness compared to log-replay traffic flows?
We conduct experiments the Waymo Open Motion Dataset (WOMD) (LLC 2019), a large-scale benchmark for motion forecasting and simulation. WOMD contains scenarios captured at 10Hz, providing second of historical data and seconds of future trajectories per scene. Each scenario includes up to 128 traffic participants (vehicles, cyclists, pedestrians) along with high-definition maps. To reduce computational cost, downsample each scenario to 2Hz, yielding 19 discrete steps per scene. We use ScenarioNet (Li et al. (2023) to manage data. SceneStreamer is trained to predict all three types of agents and all agents in the scenario. Details of hyperparameters, training and testing can be found in the Appendix.
3.1 INITIAL STATE QUALITY
To assess the realism of SceneStreamer- gennrrted initial states, we use the Maximum Mean Dis- crepancy (MMD) metrie, standard measure of distributional divergence in generative model-

---

Published as a conference paper at ICLR 2026
Table I: Initial state MMD metrics. These methods have access to future agent trajectories and use them to assist in generating initial states, making them incomparable to our setting, where the model performs state initialization without any future information. I We relax the standard evaluation protocol by computing MMD over all logged agents with arbitrary category (instead of only vehicle agents within 50m of the ego vehicle).
Method
Position
Heading
Size
Velocity
MotionCLIP
0,1236
0.1446
0.1234
0.1958
TrafficGen
0.1451
0.1325
0.0926
0.1733
LCTGen
0.1319
0,1418
0.1092
0.1948
UniGen Joint
0,1323
0.2251
0,0831
0.1915
UniGen w/ Agent-Centric Road
0.1217
0.1095
0.0817
0.1679
UniGen w/ Traj. Inputs
0.1197
0.1897
0.0826
0.1657
UniGen Combined!
0.1208
0.1104
0.0815
0.1591
SceneStreamer w/o AR Decoding
0,1603
0,1646
0,1172
0,2114
SceneStreamer
0,1291
0.1270
0,0743
0,1970
SceneStreamer w/o AR Decoding
0.3237
0.1203
0.0630
0.1183
SceneStreamer!
0.2198
0.0665
0.0279
0.0730
ing (Mahjourian et al. 2024j. Lower MMD indicates closer alignment between generated and real agents. We evaluate under two settings: 1) a strict protocol from TrafficGen (Feng et al., a (2023), con- sidering only vehicles within 50 m of the ego, and 2) a relaxed setting that includes all agents of any type (vehicle, cyclist, pedestrian), offering more comprehensive view of realism across full-scene.
We compare SceneStreamer to several recent scenario generation methods: (1) TrafficGen (Feng al., 2023); a two-stage framework generating initial states then predicting motions. (2) LCT- Gen (TTa et al.): language-conditioned scenario generator trained on natural language captions. We use the non-conditioned variant of LCTGen as in the UniGen paper. (3) MotionCLIP (Tevet itt al. 2022): diffusion- trajectory generator guided by CLIP-style embeddings, implemented LCTGen paper. (4) UniGen (Mahjourian al.) 2024): a joint model for initial state and trajec- tory generation using diffusion. (5) SceneStreamer w/o AR decoding: To evaluate the importance of SceneStreamer's autoregressive agent state decoding, we implement a simplified ablation where all agent attributes are predicted independently in parallel using separate MLP heads. Each attribute treated as a categorical variable with its own discrete space and no conditioning is performed between attributes. This resembles flat decoding strategies used in prior work (Feng et al. 2023 Tan et ) and removes the structured token sequencing that enables causally consistent agent state generation in SceneStreamer.
Table T compares SceneStreamer with recent baselines across position, heading, size, and velocity distributions. SceneStreamer achieves competitive performance, especially when using autoregres- sive (AR) decoding, under the strict evaluation protocol (vehicles only and within 50m). Under the relaxed evaluation setting (f), ScrneStreamer continues produce realistic agents beyond vehicles, baselines
demonstrating generalization to pedestrians and cyclists. Note that trajectory-infomin are not directly comparable. Methods marked use future information to refine initial states, giving them an unfair advantage over our fully predictive model. We find that SceneStreamer's 's performance drops notably when AR decoding is disabled, showing the importance of ordered token generation. Without this ordered structure, flat decoding often produces invalid combinations, e.g., pedestrian on highway lane or vehicle with inconsistent orientation and lateral velocity. Our sequential decoding mirrors the causal structure of how agents are realistically introduced into traffic scenes, improving robustness and realism in downstream simulation.
3.2 MOTION PREDICTION QUALITY
We next evaluate SceneStreamer as a motion predictor. Given the initial traffic state and agent his- tory, we autoregressively predict future trajectories of all agents over 8-second horizon. During evaluation, we state-force all agent state tokens and the first two steps of motion tokens (i.e., at f = and 0.5 seconds) and then let the model roll out the remaining steps autoregressively. We com- pare two versions of SceneStreamer: (1) ScemeStreamer-Moine that is only trained to predict motion tokens and traffic light tokens; (2) SceneStreamer-Fule The
The base version of SceneStreamer

---

Published as a conference paper at ICLR 2026
Table 2: Motion prediction metries held-out Waymo validation We evaluate Scen-
eStreamer using standard forecasting metrics for all agents and the designated object of interest (OOI).
Model
All Ageits
OOI Agouns
ADD 1
FDD
ADE
ADE 1
IDE
ADD
FDD
ADEey
ADEan
FDE
FDE
ScrmeStremmer Motion
2.2115
0.2459
1,2100
0.830
3.5336
2A4129
5.8517
0.5521
3.304
2.1905
9,7568
6.0963
ScoseStcamcc-Fut
6,3302
Real-work world Map
Motion Prodiction
Beonario Generation
Sconario Densification
Figure 4: Qualitative results of SceneStreamer in different tasks.
finetuned version of SceneStreamer-Motion that is tasked to predict all dynamic tokens. We eval- uate performance on the Waymo validation set using six standard metries: Average Displacement Error (ADE), Final Displacement Error (FDE), Average Displacement Diversity (ADD), and Final Displacement Diversity (FDD), reported for both all agents and the designated Object of Interest (OOI) defined by the WOMD.
As shown in Table 2 SceneStreamer achieves reasonable motion prediction performance. SceneStreamer--ottion provides accurate predictions with lower ADE/FDE, while SceneStreamer- Full performs slightly worse accuracy. We hypothesize this is because some attentions from the motion tokens need to be paid to the agent state tokens. Also the capability of the model might be limited due to small parameter size. However, SceneStreamer-FFil demonstrates higher ADD and FDD, indicating greater diversity.
3.3
QUALITATIVE VISUALIZATION
Fig. 4 illustrates scenes generated by SceneStreamer different settings, including motion predic- tion. full-scenario generation, and scenario densification. In the densification task, state-force the states of existing agents and ask SceneStreamer to generate new agents until 128 agents are reached. We observe that generated agents well-aligned with map lanes, exhibit coherent motion patterns, and maintain diversity over long horizons. More qualitative visualizations can be found in the Appendix.
3.4 PLANNER LEARNING WITH SCENESTREAMER
To evaluate SceneStreamer in downstream autonomous driving (AD), we train reinforcement learn- ing (RL) agents to control the self-driving (SDC) in CcenSStraamer-modiiee scenarios and test them on unaltered log-replay scenes. This setup examines whether SceneStreamer can serve as a generative simulator that improves planner robustness through diverse, reactive traffic.
8

---

Published as a conference paper at ICLR 2026
Table 3: RL policy performance trained with different traffic simulation sources.
Training Source
Reward
Success
Completion
Off-Road
Collision
Cost 4
Log- Log-Replay
32.24+3.23
0.7244
0.6726=004
0.2872+aol
0.0308-0:02
0.2852+030
SceneStreumerr Mottee (No Ada)
37.98
0,7355aau5
0.6783
0.2940auc
0.0270
0.27
ScemeStremmer-Molee (w/ Ada)
39.23+254
0,7475
0.7032=0m3
0.2987
0.0187=000
0.2637
SceneStreumer-Fe (No RS. No Ada)
38.18+3.01
0.7339
0.7052=003
0.2932+aa.
0.0194=0001
0.2697
SceneStreumer- Full (No RS, w/ Ada)
38.81
0.7385
0.7230m.m
0.3010
0.0290
0.2880
SceneStreumer-Fur Full (w/ RS, w/ Ada)
39.07
0.7620
0.7345 =0022
0,2830
0.0260=0001
0,2610
Table 4: Waymo Sim Agents Challenge (WOSAC) results on the 2025 test set leaderboard. For all metrics except minADE, higher is better.
LinSpd
LinAce
AngSpd
AngAce
DistOhj
CollLiik
DistEdge
Offread
minADE
UnIMM
[an et al. 2025
0.7822
0.3836
0.4160
0.3168
0.6491
0.3910
0.9680
0.8293
0.6791
0,9508
1.2949
CAT-K Zhinng 21256
0.7846
0,3868
0,4066
0.5201
0.6588
0,3922
0,9702
0.8302
0.6814
0.9524
1.3065
SconeSir coree
0,4030
0.4232
0.5930
0.3873
0.9694
0.6730
1.4252
We use 500 WOMD training scenarios, replacing background traffic with SceneStreamer--eneeee agents while keeping the SDC trajectory fixed (for computing reward and route completion). Train- ing runs in MetaDrive (Li al. 2022), which imports scenarios from ScenarioNet (Li et al. (023) via a unified scenario description format. We convert SceneStreamer outputs into this format, hence seamlessly integrating SceneStreamer with the RL training pipeline. Policies are trained with TD3 (Fujimoto et al. 2018) for 2M steps and evaluated on 100 held-out WOMD validation scenar- ios. We report standard RL metrics: 1) Average Episodic Reward, 2) Episode Success Rate: Fraction of episodes that terminate successfully (i.e., reaching goal without major violation), 3) Route Com- pletion Rate: Fraction of the predefined route (from GT SDC trajectory) completed per episode, 4) Off- off-RRod Rate: Fraction of episodes in which the agent deviates off off-road, 5) Collision Rate: Frac- tion of the episodes that have collisions, 6) Averuge Cost: The average number of collisions happen in one episode. Full RI. environment details are provided in the Appendix.
We compare several regimes. 1) Log-Replay is the baseline trained with unmodified roal-world traffic agents. 2) SceneStreamer uses the original initial states of background agents and generates motions background agents with SceneStreamer. generates the initial layout of all agents except SDC, rolls out the motions of background agents, and keeps adding new agents if existing agents leave scene. The variant "adaptive" (w/ Ada) means state-force SDC's trajectory using the latest RL planner's 's own rollout, otherwise (No Ada) SDC follows the ground-truth trajectory. The adaptive version enables the closed-loop training for the RL planner (Zhang et al., (023a):: the behavior of SceneStreamer will be influenced by current SDC planner and thus the generated scenarios are conditioned on current RL agent. For SceneStreamer- Full, Reject Sampling (RS) means we regenerate an agent if it collides with existing agents.
contrast, 3) Scene
Table 3 shows that SceneStreamer--eenrrace across all metrics. Even without full scenario generation, motion-only variants outperform the log- replay baseline. Adaptive training raiere the SDC follows the planner's 's rollout furthhr improves robustness and reward. The best-performing setup uses full scenario generation with reject sampling, achieving the highest route completion and lowest cost, demonstrating SceneStreamer's utility as high-fidelity simulation platform for RL policy training.
scenarios consistently improve planner performance
3.5 WOSAC RESULTS
Table reports performance the 2025 Waymo Sim Agents Challenge test set. SceneStreamer achieves competitive realism and behavioral likelihood metrics compared to strong baselines such as UniMM (Lin al.] 2025) and CAT-K (Zhang et al. (025b). which benefit from mixture-of-experts modeling and closed-loop fine-tuning, respectively. Although SceneStreamer does not outperform on minADE, it maintains strong performance across most realism metrics, validating its efficacy as a general- purpose simulator.
RELATED WORK
Motion Prediction and Simulation Agents. Motion prediction models aim to forecast future trajec- tories of traffic participants given their initial states, maps, and signals. Classical approaches model

---

Published as a conference paper at ICLR 2026
agents independently (Chai al. 2019 Shi et al.
2023 with joint interaction modeling (Luo
jet al. 2023; Wang et al. 2023). More recent transformer based models learn to autoregressivell
predict motions in open- loop or semi-closed-leee loop fashion (Kamenev et al.) 2022] SSff et al. et al. 2004). These
2023 Zhang et al. 2023b Philion et al. Hu et al. 2024 Zhou et (0244
models typically assume fixed agent set and focus only on forward rollout, without modifying the initial scene layout. SceneStreamer complements this line of work by modeling both the motion and the generative process of agent state creation, enabling adaptive and evolving agent populations during simulation.
Scenario Generation. Scenario generation aims to produce both the initial agent states and their future trajectories. Early methods adopt a two-stage design: generating static snapshots (Feng et al.
2023 Tan et al., 2021: 2023) followed by motion forecasting using separate module. While fective, such disjoint designs lack shared context across stages and restrict dynamic updates to the agent set. Diffusion-basse based approaches (Lu et al., 2024, Sun et al., 2024] Chitta et al. [024]] Diang ct al. 2024 learn scene-level generative priors using denoising processes; SceneDiffuser in par- ticular uses a single latent diffusion model for both initialization and rollout with constraint-based control, providing unified diffusion formulation for driving simulation. Closer to our approach are unified generative simulators that jointly model initial states and motions within one model. Uni- Gen (Mahjourian et al. 2024) jointly generates initial states and motions but performs generation only once at initialization and cannot inject new agents mid-simulation. Concurrent to our work, Interleaved InfGen (Yang al., 2025) uses a single autoregressive model to interleave motion sim- ulation with agent addition, removal, and (re)initialization from logged Is seed using ego-centric occupancy grids and a dynamic agent matrix. SceneStreamer instead uses map-anchored token groups shared across initialization, densification, and rollout, enabling scenario generation directly from map tokens (without access to ego state), simulaatttioon scene editing, and densification with single aenne-rrpp--alggee abstraction. Its compact discrete token-group autoregressive transformer with explicit map-anchored state and traffic-light tokens makes it practical as fast, closed-loop sim- ulator for training reinforcement-io -erarccmnne--t-innnggee.ee -ased planners. A more comprchensive review of related literature is provided in the Appendix.
5 CONCLUSION
We introduced SceneStreamer, a unified generative traffic simulator. By representing heteroge- neous traffic elements such as vehicles, cyclists, pedestrians, and traffic lights as discrete tokens, SceneStreamer enables flexible, step-by-step simulation of complex traffic scenes. This design en- ables a wide range of use cases in inference, including motion prediction, scenario densification, and synthetic scene generation, without modifyyng the model. Unlike prior methods that rely fixed initial conditions or log-replay agents, SceneStreamer supports dynamic agent injection and closed- loop rollout, facilitating long- long-horizon and reactive simulations. Shown with extensive experiments, SceneStreamer generates high-fidelity initial states, maintains coherent and diverse traffic behaviors over time, and improves the robustness and generalization of downstream RL planners trained in its generated scenarios.
Limitations. SceneStreamer relies on long token sequences represent dense multi-agent traffic scenes. This leads to high memory demand during training. Another challenge lies in compounding errors during test-time generation. We can opt for recent advances in closed-loop fine-tuning
behavior model to address this issue (Zhang et al. 2025b Peng et al. 2024 Chen et al. 2025).
ACKNOWLEDGMENTS
This project was supported by the NSF Grants CNS-2235012 and CCF-2344955.
REFERENCES
Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Blakej Osiński, Hugo Grimmett, and Peter Ondruska. Simnet: Learning reactive self-driving simulations from real-world observations. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 5119-5125. IEEF, 2021.
10

---

Published as a conference paper at ICLR 2026
Axel Brunnbauer, Luigi Berducci, Peter Priller, Dejan Nickovic, and Radu Grosu. Scenario- bassd curriculum generation for agent autonomous driving. arXiv preprint FrXiv:2403.17860 2024.
Yulong Cao, Boris Ivanovic, Chaowei Xiao, and Marco Pavone. Reinforcement learning with human feedback for realistic traffic simulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14428-14434. IEEE, 2024.
Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple prob- abilistic anchor trajectory hypotheses for behavior prediction. arXiv preprint arXiv:1910.0544 2019.
Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chandraker. Safe-sim: Safety-critical closed-loop traffic simulation with diffusion-controllabbe adversaries. In European Conference on Computer Vision, pp. 242-258, Springer, 2024.
Keyu Chen, Wenchao Sun, Hao Cheng, and Sifa Zheng. Rift: Closed-loop rl fine-tuning for realistic and controllable traffic simulation. arXiv preprint wxxv:2505.033-44 2025.
Kashyap Chitta, Daniel Dauner, and Andreas Geiger. Sledge: Synthesizing driving environments with generative models and rule-based traffic. European Conference on Computer Vision, pp- 57-74. Springer, 2024.
Wenhao Ding, Yulong Cao, Ding Zhao, Chaowei Xiao, and Marco Pavone. Realgen: Retrieval augmented generation for controllable traffic scenarios. European Conference on Computer Vision, pp. 93-110. Springer, 2024.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pp. 1-16, 2017.
Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolei Zhou. Trafficgen: Learning gen- erate diverse and realistic traffic scenarios. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 3567-3575, IEEE, 2023.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018.
http:htpppppe httt:::/proceeeingnge wwt.mm....ppssse.. (ttess//80/Eujinncee
html
Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile trollability. In The ehirth- Annual Conference on Neural Information Processing Systems.
Cole Gulino, Justin Fu, Wenjie Luo, George Tucker, Eli Bronstein, Yiren Lu, Jean Harb, Xinlei Pan, Yan Wang, Xiangyu Chen, et al. Waymax: An accelerated, data-driven simulator for large- scale autonomous driving research. Advances Neural Information Processing Systems, 36: 7730-7742, 2023.
Zhiming Guo, Xing Gao, Jianlan Zhou, Xinyu Cai, and Botian Shi. Scenedm: Scene-level multi- agent trajectory generation with consistent diffusion models. arXiv preprint arXiv:2311.15733 2023.
Yihan Hu, Siqi Chai, Zhening Yang, Jingyu Qian, Kun Li, Wenxin Shao, Haichao Zhang, Wei Xu, and Qiang Liu. Solving motion planning tasks with a scalable generative model. In European Conference on Computer Vision, pp. 386-404. Springer, 2024.
Chiyu Jiang, Andre Comman, Cheolho Park, Benjamin Sapp, Yin Zhou, Dragomir Anguelov, et al. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. Proceedings of the IEFE/CV conference computer vision and pattern recognition, pp. 9644 0654-9653, 2023.
11

---

Published as a conference paper at ICLR 2026
Max Jiang, Yijing Bai, Andre Commaan, Christopher Davis, Xiukun Huang, Hong Jeon, Sakshum Kulshrestha, John Lambert, Shuangyu Li, Xuanyu Zhou, et al. Scenediffuser: Efficient and con- trollable driving simulation initialization and rollout. Advances in Neural Information Processing Systems, 37:55729-55760) 2024.
Alexey Kamenev, Lirui Wang, Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nistér, and Nikolai Smolyanskiy. Predictionnet: Real-time joint proba- bilistic traffic prediction for planning, control, and simulation. In 2022 International Conference on Robotics and Automation (ICRA), pp. 8936-8942. IEEE, 2022.
Saman Kazemkhani, Aarav Pandya, Daphne Comelisse, Brennan Shacklett, and Eugene Vinit- sky. Gpudrive: Data-driven, multi-agent driving simulation at million fps. arXiv preprint arXiv:2408.01558, 2024.
Edouard Leurent. An environment for autonomous driving decision decision-making. https:///ithhi
Com/ hom/eelae wwewrent/niggwwayaeee
2018.
Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified decupancy-centrre driving scene gener- ation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 11971 11981, 2025.
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 2022.
Quanyi Li, Zhenghao Mark Peng, Lan Feng, Zhizheng Liu, Chenda Duan, Wenjie Mo, and Bolei Zhou, Scenarionet: Open-source platform for large-scale traffic scenario simulation and model- ing, Advances in neural information processing systems, 36:3894-3920, 2023.
Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang, Rong Xiong, and Yue Wang- Revisit mixture models for multi-agent simulation: Experimental study within a unified frame- work. arXiv preprint arXiv:2501. 17015, 2025.
Waymo LLC. Waymo open dataset: An autonomous driving dataset, 2019.
Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker- Bieker-Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamaric Wießnnrr Microscopic traffic simulation using sumo. In 2018 21st international conference on intelligent transportation systems (ITSC), pp. 2575-2582. Iece, 2018.
Jack Lu, Kelvin Wong, Chris Zhang, Simon Suo, and Raquel Urtasun. Scenecontrol: Diffusion for controllable traffic scene generation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 16908-16914. IEEE, 2024.
Wenjie Luo, Cheol Park, Andre Cormman, Benjamin Sapp. and Dragomir Anguelov. Jfp: Joint future prediction with interactive multi-agent modeling for autonomous driving. In Conference on Robot Learning, pp. 1457-1467. PMLR, 2023.
Reza Mahjourian, Rongbing Mu, Valerii Likhosherstov, Paul Mougin, Xiukun Huang, Joao Mes- sias, and Shimon Whiteson. Unigen: Unified modeling initial agent states and trajectories for generating autonomous driving scenarios. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 16367-16373. IEEE, 2024.
Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jef- frey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene trans- former: A unified architecture for predicting future trajectories of multiple agents. In Interna- tional Conference on Learning Representations.
Zhenghao Peng, Wenjie Luo, Yiren Lu, Tianyi Shen, Cole Gulino, Ari Seff, and Justin Fu. Improving agent behaviors with fine-tuning for autonomous driving. In European Conference on Computer Vision, pp. 165-181. Springer, 2024.
12

---

Published as a conference paper at ICLR 2026
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jonah Philion, Xue Bin Peng and Sanja Fidler. Trajeglish: Traffic modeling next-token predic- tion. In The Twelfth International Conference on Learning Representations.
Ethan Pronovost, Meghana Reddy Ganesina, Noureldin Hendy, Zeyu Wang. Andres Morales, Kai Wang, and Nick Roy. Scenario diffusion: Controllable driving scenario generation with diffusion. Advances in Neural Information Processing Systems, (1168873-68944, 2023.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL
1364.364... m1mml
Davis Rempe, Jonah Philion, Leonidas Guibas, Sanja Fidler, and Or Litany. Generating useful accident-prone driving scenarios via a learned traffic prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17305-17315, 2022.
Luke Rowe, Roger Girgis, Anthony Gosselin, Bruno Carrez, Florian Golemo, Felix Heide, Liam Paull, and Christopher Pal. CtRL-sim: Reactive and controllable driving agents with offline reinforcement learning. In 8th Annual Conference on Robot Learning, 2024. URL https: 7/openzeview wttt/foruuue dd
Luke Rowe, Roger Girgis, Anthony Gosselin, Liam Paull, Christopher Pal, and Felix Heide. Sce- nario dreamer: Vectorized latent diffusion for generating driving simulation environments. arXiv preprint arXiv:2503..m 250.:2296... 2025.
Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Re- faat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 8579-8590, 2023.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 35:6531-6543, 2022.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. arXiv preprint arXiv:2306.1777 2023.
Shuo Sun, Zekai Gu, Tianchen Sun, Jiawei Sun, Chengran Yuan, Yuhang Han, Dongen Li, and Marcelo H Ang. Drivescenegen: Generating diverse and realistic driving scenarios from scratch. IEEE Robotics and Automation Letters, 2024.
Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi- agent behaviors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10400-10409, 2021.
Simon Suo, Kelvin Wong. Justin Xu, James Tu, Alexander Cui, Sergio Casas, and Raquel Urtasun. Mixsim: A hierarchical framework for mixed reality traffic simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9622-9631. 2023.
Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Krachenbuchl. Language conditioned traffic generation. In 7th Annual Conference Robot Learning.
Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. In Proceedings the
the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 892-901, 2021.
13

---

Published as a conference paper at ICLR 2026
Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Krachenbuehl. Language conditioned traffic generation. In Conference on Robot Learning. pp. 2714-2752. PMLR, 2023.
Shuhan Tan, John Lambert, Hong Jeon, Sakshum Kulshrestha, Yijing Bai, Jing Luo, Dragomir Anguelov, Mingxing Tan, and Chiyu Max Jiang. Scenediffuser++: City-scale traffic simulation via a generative world model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1570-1580, 2025.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or: Motionclip: Ex- posing human motion generation to clip space. In European Conference on Computer Vision, pp- 358-374. Springer, 2022.
Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, and Jakob Foerster. Nocturne: scalable driving benchmark for bringing multi-agent learning one step closer to the real world. Advances in Neural Information Processing Systems, 35:3962-3974, 2022.
Sheng Wang, Ge Sun, Fulong Ma, Tianshuai Hu, Qiang Qin, Yongkang Song, Lei Zhu, and Junwei Liang. Dragtraffic: Interactive and controllable traffic scene generation for autonomous driving. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 14241- 14247. IEEE, 2024.
Yu Wang, Tiebiao Zhao, and Fan Yi. Multiverse transformer: 1st place solution for waymo open sim agents challenge 2023, arXiv preprint arXiv:2306.11868 2023.
Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: Scalable multi-agent real-time motion generation via next-token prediction. Advances in Neural Information Processing Systems, 37:
114048-J 1 114071, 2024.
Xiuyu Yang, Shuhan Tan, and Philipp Krähenbühl. Long-term traffic simulation with interleaved autoregressive motion and scenario generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 25305-25314, 2025.
Xuemeng Yang. Licheng Wen, Yukai Ma, Jianbiao Mei, Xin Li, Tiantian Wei, Wenjie Lei, Daocheng Fu, Pinlong Cai, Min Dou, Botian Shi, Liang He, Yong Liu, and Yu Qiao. Drivearena: A closed- loop generative simulation platform for autonomous driving. arXiv preprint WrXiv:2408.00415. 2024.
Linrui Zhang, Zhenghao Peng, Quanyi Li, and Bolei Zhou. Cat Closed-loop adversarial training for safe end-to-end driving. Conference on Robot Learning, pp. 2357-2372, PMLR, 2023a.
Qichao Zhang, Yinfeng Gao, Yikang Zhang, Youtian Guo, Dawei Ding, Yunpeng Wang, Peng Sun, and Dongbin Zhao. Trajgen: Generating realistic and diverse trajectories with reactive and fea- sible agent behaviors for autonomous driving. IEEE Transactions on Intelligent Transportation 2022.
Systems, 23(12):
Shenyu Zhang, Jiaguo Tian, Zhengbang Zhu, Shan Huang, Jucheng Yang, and Weinan Zhang. Drivegen: Towards infinite diverse traffic scenarios with large models. arXiv preprint arXiv:2503.05888, 2025a.
Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Lue Van Gool. Trafficbots: To- wards world models for autonomous driving simulation and motion prediction. 2023 IEEE IEEE, 2023b.
International Conference Robotics and Automation (ICRA). pp.
Zhejun Zhang, Christos Sakaridis, and Lue Van Gool. Trafficbots vl. 5: Traffic simulation via conditional vaes and transformers with relative pose encoding. arXiv preprint arXiv:2406. 10898, 2024.
Zhejun Zhang, Peter Karkus, Maximilian Igl, Wenhao Ding, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Closed-loop supervised fine-tuning of tokenized traffic models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2025b.
Jianbo Zhao, Jiaheng Zhuang, Qibin Zhou, Taiyu Ban, Ziyao Xu, Hangning Zhou, Junhe Wang, Guoan Wang. Zhiheng Li, and Bin Li. Kigras: canematic-drivenn generative model for realistic agent simulation, 2024. URL htt.sttts://///mmmmmm http://///..aaaooooo////////////
14

---

Published as a conference paper at ICLR 2026
Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-guider traffic simulation via scene-level diffusion. Conference Robot Learning, pp. 144-177. PMLR, 2023a.
Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp- 3560-3566. IEEE, 2023b.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Mont- gomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang. Ying Wen, Kimia Hassan- zadeh, Daniel Graves, Dong Chen, Zhengbang Zhu, Nhat Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiamnan Wu, Zhengang Fu, Kasra Rezaee, Peyman Yadmel- lat, Mohsen Rohani, Nicolas Perez Nieves, Yihan Ni, Seyedershad Banijamali, Alexander Cowen Rivers, Zheng Tian, Daniel Palenicek, Haitham bou Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, and Jun Wang. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving, 2020.
Zikang Zhou, Jiamping Wang, Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 17863-17873, 2023.
Zikang Zhou, HU Haibo, Xinhong Chen, Jianping Wang, Nan Guan, Kui Wu, Yung-Hui Mui Li, Yu-Kai Huang, and Chun Jason Xue. Behaviorgpt: Smart agent simulation for autonomous driving with next-patch prediction. Advances in Neural Information Processing Systems, 37:79597-79617, 2024.
15

---

Published as a conference paper at ICLR 2026
Appendix
APPENDIX CONTENTS
A. Broader Impact and Societal Considerations
Exteneed Related Work
B.1 Motion Prediction and Simulation Agents
B.2 Scenario Generation
B.3 Data-Driven Simulation
C. Model Architecture Details
Encoder-Decoder Structure
C.2 Token Embeddings and Types
C.3 Relative Positional Attention
C.4 Token Group Attention Mechanism
Agent State Tokenization
Training and Inference Details
E. Dataset and Preprocessing
Tokenization Hyperparameters
E.3 Model Training and Inference Details
E.4 Reinforcement Learning Setup
F. Frequently Asked Questions
G. Qualitative Visualizations
A BROADER IMPACT AND SOCIETAL CONSIDERATIONS
SceneStreamer is generative simulation framework for modeling dynamic traffic scenarios using autoregressive token group prediction. By enabling realistic, reactive, and scalable traffic simula- tion, SceneStreamer has the potential to significantly advance the development and validation of autonomous driving (AD) systems. This includes improving the robustness motion planning poli- cies as we studied in the experiment section, facilitating rare event training, and supporting data augmentation in reinforcement learning pipelines.
Positive Societal Impacts. SceneStreamer's 's ability to generate diverse and reactive traffic scenes accelerate the safe deployment of AD systems. More robust planners may reduce traffic cidents, improve traffic efficiency, and enhance accessibility for populations with limited mobility. Furthermore, open-sourcing model and implementation encourages broader research into safety- critical domains without requiring access to expensive real-world data collection or proprietary plat- forms.
Potential Negative Impacts and Misuse. As a scenario generation tool, SceneStreamer could be misused to simulate rare or malicious driving scenarios for purposes such as adversarial testing without disclosure or crafting unfair benchmarks. Additionally, if used to train agents without proper safety constraints, generated scenarios might lead to overfitting to synthetic patterns or unsafe gen- eralization in deployment. There is also potential for use in generating deceptive traffic scenes in virtual testing or regulatory submissions.
Mitigations. We emphasize that SceneStreamer is not a closed-loop SDC driving policy and does not dictate real-world behavior. However, we encourage the community to adopt responsible use practices. This includes transparent reporting of synthetic data usage, gating scenario difficulty and validity when used in SDC planner training and evaluation, and coupling SceneStreamer with validation on real-world data. Our open-source release will include documentation clarifying its intended research and limitations.
16

---

Published as a conference paper at ICLR 2026
B EXTENDED RELATED WORK
B.I MOTION PREDICTION AND SIMULATION AGENTS
Motion prediction models aim forecast future trajectories of traffic participants given their past states, road maps, and traffic signals. Classical approaches often treat agents independently (Shi
al. 2022 Chai et al. 2019: Shi et al. 2023 Wang al., 2024). while more recent models incor-
porate joint interaction modeling (Luo et al., 2023 Wang et al.) 2023 Ding et al. 2024] Suo et al., Zhang et al.] [022] 2023b
2021
2024). Transformer- based models have further ad advanced this field
by learning to autoregressivell predict motions in open-loop or semi-closed loop setups (Kamenev
cet al.
Seff et al. [2023] Philion et al., [Hu et al. 2024; Zhou et al. 2024 Zhao et al.. 2024| Lin 2025 Zhang et al. 2025b). In parallel, diffusion models have been introduced an alterna- tive generative paradigm, including MotionDiffuser (Jiang et al., 2023) and SceneDM (Guo et al,
2022
ct al.
Chang et al. 2024). Despite impressive progress, these methods operate under the assumption that agents set is fixed and focus solely on the trajectory rollout. They do not modify or expand the initial scene configuration, limiting their use in dynamic simulation. SceneStreamer addresses this limitation by jointly modeling both agent state generation and motion rollout, supporting dynamic agent populations during long-horizon simulation. Furthermore, because they require full access to past and current states for all agents, these motion models are not directly applicable to the scenario generation setting.
2023
B.2 SCENARIO GENERATION
Scenario generation involves synthesizing both initial conditions and future evolutions of traffic scenes. Procedural generation approaches (Li et al. 2022 Lopez et al. 2018 Dosovitskiy et al.
2017 Zhou et al. 2020: Leurent 2018; BBrunnbauer et al. 2024) rely on hand-coded rules or tem-
plates, which limits realism and diversity. Many lcarning-basee works adopt a two-stage pipeline:
static scene generation followed by motion forecasting (Feng et al. 2023 Tan al. 2021: 2023
Cao et al. 2024, Pronovost et al. 2023 Bergamini et al. 2021). For example, SceneGien CTan jet al. 2021 and TrafficGen (Feng et al.) 2023) autoregressively add agents based map anchors,
followed by state refinement. SceneStreamer builds on this idea but unifies state and trajectory generation into single model, promoting global consistency and flexible editing. Diffusion-basse
methods have also been proposed for full-scene generation (Lu et al. 2024 Sun et al. 2024] Chitta al. 2024. Rowe et al. (202)) including CTG (Zhong et al.] [202bb a) and SceneDiffuser (Jiang et al. 2024 Tan et al.]|202,, 2025) which generate dense scenes under language guidance. However, some of these models still separate static and dynamic phases, or generate entire scenes in single forward pass, limiting interactivity. SceneDiffuser in particular uses a single latent diffusion model for both initialization and rollout with constraint- -assd control, providing a unified diffusion formulation for driving simulation.
UniGen MMajourraan et al. 2024) improves on prior work by jointly generating initial states and motion trajectories. However, it generates scenes in fixed order. UniGen first initializes agent A's state, then predicts the full future trajectory of agent A. Then it initializes the agent B s' state, while agent A's future trajectory is accessible. This breaks temporal causality and creates difficulty if we want to conduct closed-loop simulation with it. Moreover, its agent-centric representation requires expensive replanning maintain closed-loop consistency. In contrast, SceneStreamer generates agent states and motions in a unified token sequence using a single autoregressive model. This enables realistic, causal interactions and allows agents to enter or leave the scene dynamically.
Concurrent to our work, Yang et al. propose InfGen (Yang et al. 2025). which uses a single autore- gressive model to interleave motion simulation with agent addition. removal, and (re)initialization. It discretizes agent poses on an ego-centric occupancy grid and expands a dynamic agent matrix with control tokens starting from logged 1s seed. SceneStreamer differs from these works in sev- eral ways. First, it uses map-anchored token groups shared across initialization, densification, and rollout, rather than ego-centric occupancy grids and pose tokens. This provides a single, lane-graph- aligned abstraction for both initial states and motions and supports scenario generation directly from map tokens, as well as simulatiuttion scene editing and densification by injecting new agents while keeping the same representation. Second, SceneStreamer is designed as compact, discrete token- group autoregressive transformer with explicit map-anchored state and traffic-light tokens, enabling direct token-level editing via ftrre-foreing and resampling. This lightweight design makes it practi-
17

---

Published as a conference paper at ICLR 2026
cal to fast, closed-loop simulator for training ceafforremeet-leammin based planners, where we show that training on SceneStreamer SceneStreamer--eeneatee scenarios improves robustness and generalization over log-replay baselines.
B.3 DATA- DATA-DRIVEN SIMULATION
Data-driven simulation environments such Nocturne (Vinitsky al. 2022). Waymax (Gulino et all. 2023). MetaDrrve (Li et al. 2022). ScenarioNet (Li et al. (023). and GPUDrive (Kazemkhani al. 2024) enable scalable simulation by replaying real-world logs. While preserving behavioral realism, the traffic flows in these environments are non-reactive: deviations from the logged trajec- tory, e.g., when the ego vehicle brakes earlier, can result implausible interactions like rear-end collisions. Recent advances integrate generative models to create reactive and closed-loop simula- tion environments. Vista (Gao et al.) predicts future higo-resouttion images and supports interactive control, while DriveArena (Yang et al. 2024) combines neural renderer with physics-basee simulator, forming tight perception-action
loop. UniScene (Li et al., (025) introduces a unified occupancy-centric framework that first generates semantic occupancy from BEV layouts and then conditions on to synthesize multi-view video and LiDAR, enabling versatile and high-fidelity scene generation. These approaches focus photorealistic sensor simulation, helping bridge the sim-to-real gap for perception modules.
In terms of interaction-level simulation, works like STRIVE (Rempe al. 2022) and CAT (Zhang et al., (223a) generate safety-critical scenarios for safety validation. MixSim (Suo et al., 2023 goal-conditioned policy and actively resimulate different possible goals enable closed-loop simulation, but its computational cost scales poorly with the number of agents, limiting real-time use. CtRL-Sim (Rowe et al. 2024) applies offline RL to train reactive agents for use in Nocturne, enabling goal-directed, controllable traffic behavior. SceneStreamer complements these works by acting fast, flexible scenario generation model that not only generates trajectory but also initial- izes new agents.
C MODEL ARCHITECTURE DETAILS
This section presents the details of SceneStreamer: a unified transformer framework that jointly generates traffic-light states, agent initial states, and agent motions in single autoregressive token sequence.
Our Insights. We cast scenario generation as next-token prediction task: map tokens (<MAP>> are followed each step by traffic-light tokens <TL>, agent-state tokens <AS>> and agent-motion tokens <MO> and form single autoregressive token sequence:
X CCMAP>;<TLL>> <AS>,
<MO>) (<)T>))((TT))))..)))) <ATL> <AS>, <MO>)..)))
Given all tokens X<t generated so far, the model predicts the categorical distribution the next token pa(#; xx1) and samples 2;. Following this idea, develop SceneStreamer learning one transformer that sees the whole history, enabling -raiied closed-loop generation and smoother downstream RL integration.
18

---

Published as a conference paper at ICLR 2026
Map
Fugh's
The
Agone ypu!
Mapp Ib
Fhebattve
ArOS
MON
Tokens
Agent Tokens
HeadTL
HeadType
HeadMengit
Falattvv State Hoad
HeadMO
Condition
Encoder
SceneStreamer Decoder
Map
Maps
Thit
A5
SDSt
Agent Typa
Mapp
A.
ArOS
MO
Figure 5: SceneStreamer model architecture.
C.1 EXCODER-DECODER STRUCTURE
SceneStreamer adopts the encoder-decoder architecture. A lightweight encoder embeds up to 3000 map-segment tokens (produced by slicing each lane-centerline into < 10 segments) into a set of key/value vectors HuuP. The output map tokens are later used for cross attention. A decoder autoregressively generates all non-map tokens. In every layer of the decoder, we first conduct self- attention within the input token sequence, where a group attention causal mask illustrated in Fig. applied. Then we conduct cross-attention between the dynamic tokens and the map tokens.
Prediction Heads. As shown in Fig. 5 SceneStreamer uses a shared decoder trunk followed by distinct output heads for each token group. Each head projects the decoder hidden state task- specific vocabulary or output space.
(I) Traffic light head is an MLP layer HeadTL((<TL>x)) Mm, e IR x4 maps the decoder output to one of four discrete states: fgreen, yellow, red, unknown).
(2) Agent State Head a nested module with several sub-heads and an agent state transformer. We will discuss this in appendix D Overall, in the agent state generation, two MLPs (the agent type prediction head HeadType and the map ID) predictor HeadMapIDD as well as tiny transformer, the Relative State Head, involved.
(3) Motion Head: The motion head predicts each agent's control input as a single categorical token from 2D discretized space of acceleration and yaw rate. Specifically, we define a flat vocabulary, where each token corresponds to unique pair (a, w) drawn from uniformly quantized grids A and Q. We apply a single linear classifier: HeadMO( (1M00<<>>,)) E R1,089 followed by softmax layer to obtain the probability distribution over control tokens. At inference time, we decode the token index using nucleus sampling and map back the corresponding (a, w) pair via deterministic lookup table. The predicted control input is then passed through the kinematic update rule to compute the agent's new state.
Each prediction head operates only on its associated tokens, enabled by token-type embedding and mask within the decoder. This modular structure allows SceneStreamer to handle heterogeneous outputs while maintaining unified sequence modeling.
C.2 TOKEN EMBEDDINGS AND TYPES
Map Tokens <MAP>. A map segment is represented as polyline, consisting of N, ordered 2D points with semantic attributes. We denote the set of M map segments in the scene as a tensor Smap E MxxxC where C is the per-point feature dimension (c.g.. 2D position, road type one- hot). We adopt the PointNet-like (Qi et al. 2017) polyline encoder yielding map segment features (p: PolyEnc(S!0) M
These are then passed into the SceneStreamer encoder with full self-
19

---

Published as a conference paper at ICLR 2026
attention across map segments:
Tmap SceneStreamerrc :Parl))) Rxdd
To support cross-attention in the decoder, we assign each map segment a unique discrete index i (its MapID), and embed it into the map token.
<MP
[t] + EmbMapID(1) @ gg. L...., M..
(8)
where EmbMapID is a learned embedding table. @ denotes we will record the geometric information g; of map segment which includes its center position and heading, and use participate in the relative attention. We defer the discussion of relative attention to appendix C.3
These enriched map tokens {<MAP>>} are kept fixed during simulation and serve static cross- attention keys/values for all decoder layers. Each dynamic token (c.g., <TL>, <AS>, <MO>> performs coss-attention to the map encoder output to incorporate geometric context.
Traffic Light Tokenn <TL>. Each traffic light is represented by single token per step. We encode the traffic light's discrete state (green, yellow, red, or unknown), its unique identifier, and the map segment it resides in Ag. Formally, the traffic light token for light at step is constructed as:
<TL>RR (mbStaate(w.t)) + EmbTLID(k) EmbMapID(Aal & gk; 1, y.. NTL.
where sh, E YG, U} is the signal state, 0g is the discrete map segment ID the light is attached to, and ge is its temporal-gcometri context (position, orientation and current timestep). As with map tokens, & indicates that g& participates in relative attention (see appendix C.3). All traffic light tokens are generated in a single batch at each step, with the output obtained via 4-way classification head.
Agent State Tokens <AS>. For every active agent-including newly injected agents at step --ScenSStreamer generates set of four agent state tokens that collectively encode the agent's ' dynamic and semantic state. These states include positions, headings, velocities, shapes and agent categories. We defer the detailed tokenization and inference process of agent state tokens to ap- pendix D
Motion Tokens <MO>. To model agent motion, SceneStreamer predicts tokenized instanta- neous control input for each agent, parameterized as pair of acceleration and yaw rate: (a, w) E A Ω (Zhao et al., 2024), where and I2 are discretized into 33 uniform bins respectively, covering acceleration and yaw rate ranges observed in training data. This results in total of 33 x 33 = 1,089 motion classes. Given an agent's current state time position (1), heading the, and speed tt, the next-step state is predicted using first-order bicycle-model update over small timestep At (we use At 0.5s):
V++1 = 1/2 + w Al,
(10)
U141 + a At,
(11)
F++1 = I; + 0++1 (V5(1))11))))) At,
(12)
Ut+1 Ye 0++1 (1 AL.
(13)
We assume zero lateral slip and no wheelbase constraint (i.e., velocity direction aligns with heading).
To obtain the ground- -ruth motion token for agent i at step we enumerate all 1,089 candidate (a,w) combinations, apply the above update rule to generate candidate next poses, and evaluate them against the true bounding box L + 1. Specifically: (1) For each candidate motion, compute the predicted pose (I++1, 3t++1, 1. ). (2) Generate the 4 corners of the agent's 's oriented bounding box based on its shape and predicted pose. (3) Compute the Average Corner Error (ACE) as the mean & distance between predicted and ground-truth corners. (4) Select the (a, w) pair minimizing ACE as the ground- -ruthh label 14,
This strategy ensures that both position and heading tightly aligned during supervision. Com- pared to velocity- or displacement- basse tokenization schemes (Wu et al., 22024] Philion et al.). our control-based formulation provides smoother interpolation of motion intent and better supports
20

---

Published as a conference paper at ICLR 2026
maneuver modeling such as lane changes and turns. It also enables compact tokenization with high spatial precision.
Each motion token <MO> corresponds to agent at specific timestep and encodes both its control input and identity-related context. Formally, given an agent at step t, we define its motion token embedding
1t EmbMotionn(u)) + EmbType(c)) + EmbAID( EmbVel(v;) + EmbShape(s.) @ gi
(14) where 14, is the GT motion label selected from 1,089 candidates, C2 is the categorical for agent type (e.g., vehicle, pedestrian, cyclist), i is agent's ID, V; is a 2D vector representing the agent velocity in local frame, and s; is a 3D vector of agent's shape (length, width, height). gi is 4D vector encoding temporal-geomette step). At an agent's first appearance, a special label Psan used to get <MO>. For continuing agents, the input motion token is simply the token with previously predicted motion label f4,1-1- This enriched representation ensures that motion tokens carry sufficient context for the decoder to generate informed predictions reepitiions-capturiinn both semantic (who the agent is) and physical (how it moves) characteristics. The geometric context g, also enables relative attention with map and agent tokens, discussed in appendix C.3
information (agent's current global position, heading and time
C.3 RELATIVE POSITIONAL ATTENTION
Let By E IRd be two input tokens in Transformer layer, where token z; attends to token zy. Let their geometric temporal relation be denoted (Arij, Ayj Ariy, Algy), computed from their respective spatial anchors and time indices.
In the attention mechanism, compute the following projections:
q: (a.)) d. MI.P (I:), Fiy ML. (Aryy Agii Av/i:: Alij),
kj (z)), by = MLPv(I,),
(15)
(16)
where 9:/g/. ky. v/ &d are standard content-based query, key, and value vectors, while riy E Rd encodes the relation- aware components.
The final attention score is computed as:
1 Oi Vd
ky + + ris) + mij,
(17)
where ma E (-00,00 0} is an attention mask determined by causal constraints and group-level at- tention rules (see Fig. (). This formulation introduces spatial-tempont query to attend differently depending its learned relation to the key, improving inductive bias and
awareness by allowing each
facilitating structured interactions in traffic scenes (Zhou et al. 2023 Shi et al. 2023 Wu et al, 2024).
KNN pruning for scalable attention. To improve scalability in large scenes, we optionally apply K- K-nearest neighbor (KNN) masking attention if both query and key tokens carry relative po- sitional information. Specifically, when both tokens are equipped with geometric anchors gi and gj, we compute Euclidean distance in their x-y position and retain only the top-k closest keys for each query. This reduces the attention cost from ((N2) to O(Nk), while still preserving local in- teractions that matter for driving behavior. For tokens lacking spatial grounding (e.g., <SOS> or <TYPE>>), full attention is retained.
21

---

Published as a conference paper at ICLR 2026
Tokens at 1=0
Tokens at t=1
Traffic Light Tokens
Agent Takens
CanartL
Sell ottercin
Motion Tokens
Tamer
Cammns
Salf artenoon
TUhaaao
Metir Hartey
Sal-sbart
Tokers
Neaor trstoyy
Cenerd
But bui-ubanuun
OOO
ASMayky
Neaen Hittoyy
Figure 6: The attention mechanism in SceneStreamer. Tokens at each step are grouped into tratt (purple), agent-state (blue), and motion (green) tokens. Within timestep. attention flows from earlier groups to later groups, enforeing semantic causality. Cross-timestep attention allows history tokens to influence current predictions. Empty regions represent masked attention.
C.4 TOKEN GROUP ATTENTION MECHANISM
As shown in Fig. within the same group can attend to each other freell (e.g., motion tokens attend to other motion tokens the same step). (2) The tokens belong to the same object (agent traffic light) in later step can attend to the tokens belonging to the same object earlier. (3) Every group of token can attend to some existing contexts, for example <MO> attend current <TL>. Figure
enforce structured inter-step attention via causal group mask: (I) Tokens
6 illustrates the structured attention mask applied in the decoder. Each quadrant corresponds to token group at timestep / 0 or within each group. while the off-diagonal regions encode allowed causal flows across groups. For example, at 1, motion tokens can attend to agent-state and traffic-light tokens from both and = 1, but not vice versa. This reflects the natural temporal and semantic ordering in generative traffic scenes and helps enforce proper dependency structure during autoregressive decoding.
attending other tokens. The diagonal blocks represent full self-attention
D AGENT STATE TOKENIZATION
As shown in Fig. A(),, each agent i present step t is represented by four ordered tokens (<SOA>;, <TYPE>; <MS>;, <RS>>;)-
Here <SOA> is the of flag, <TYPE> is the categorical token in (veh, cyc, ped]. <MS> the index of map segment, <RS> is the relative states agent w.r.t. the selected map segment.
Concretely,
<SOA> EmbIntra 41) + EmbAID( + EmbSOA,
(18)
<TYPE> Embintra 4i 1) EmbAID(i) + EmbType(c:),
(19)
<MS> EmbIntra 4i + 2) EmbAID (i) + EmbType(c)) + EmbMapID(AA) & g( <(<<MAP>>)
(20)
<RS> Embintra + 3) EmbAID (i) Embbype(cc) + EmbMapID(AA) + EmbRS (r. gi.
(21)
EmbIntra (8 + j) encodes the intra-step offset of the j-th token within agent i's group, EmbAID(1)
provides consistent agent identity embedding reused across steps, EmbType(c) represents the
22

---

Published as a conference paper at ICLR 2026
(A) Agent Tovarra
(8) Map Token
Mar
(C) Falutive Stateo
where
Bupies Delature
Saadiinss exyraming 104)
Rebatie Shdee
conainn:
Save
50000032
228 30
Head
Typa
Map
Grasel
Map
SceneStreamer
Auti Murooiinyyyy prodect agen cade sivve to th map region
Agun TiPS
Mopui!
Rebadte
Figure 7: The design of agent state generation. (A) Each agent's state is encoded as 4 tokens. We first predict the agent type, select map ID where the agent resides, then predict the relative states. (B) Before obtaining the agent state, we first select a map segment as the "anchor" where the agent should reside. (C) Feeding in the Map ID, use the output token as the condition and call the Relative State Head, which is a tiny transformer, to autoregressively generate the relative agent states, including shape, position, heading and velocity.
agent's semantic class, EmbMaalID(A)) embeds the discrete map segment index X2 EmbRS(r;) embeds the agent's 's relative state r, including position, heading and velocity offsets with respect to the selected map segment and the agent's shape, g(<MAP>>.) retrieves the geometric anchor (position, heading and current step) of the selected map segment, which participates in relative attention, g, denotes the generated agent's current temporal-geometric information.
As shown in Fig. (B), by reading <TYPE>, the model will select one of the map segments X;- This is done by applying map ID head on the output token and conducting softmax sampling on the output logits:
A ~ SeftraaffeeaaapppppSSSSSeeeeeeee
(22)
As illustrated in Fig. 7 (C), after selecting a map segment A and generating the associated <MS> > token, we condition on the decoder output of <MS> to generate the agent's 's full kinematic and shape attributes. Specifically, a dedicated module called the relative state head, a small Transformer de- coder with AdaLN (Perez et al. 2018) normalization, is used autoregressively generate sequence of 8 tokens (preceded by a SOS token), each representing a field in the agent's relative state vector:
r (SOS, 1, w. h, u, U, 8rb, U2, fy),
(23)
where SOS is the start-of-sequencee indicator, (1, w. h) is the agent's physical dimensions (length, width, height), (u. 2) is the longitudinal and lateral offset from the centerline of map segment Ar dnb is the heading residual relative to the segment orientation, (Ua,, vy) is the velocity whose direction is in the frame of the map segment. Each field is discretized into 81 uniform bins and modeled classification problem.
We should mention that there are two special tokens as shown in Fig. [ the "agent state generation starts" and "agent state generation ends" token, before and after the agent state generation of all agents.
During training, teacher forcing is used to feed ground-truth relative state tokens, while at inference, we apply softmax sampling decode each dimension sequentially. Within the relative state head, the input at each decoding step consists of the embedding of last selected action out of the vocabu- lary, added to learned positional embedding that encodes its index in the sequence. This structure enables fully autoregressive decoding over the 9-token relative state sequence. Unlike prior meth- ods such as TrafficGen (Feng et al., 2023), which generate all agent state attributes simultaneously in flat and unstructured output head, SceneStreamer decomposes the generation into an ordered, interpretable sequence. This is critical for ensuring semantic and physical consistency. Specifically: Agent type must be sampled first, as it determines downstream constraints map segment valid- ity, shape bounds, and behavior priors. Map segment selection follows, as it anchors the agent in the environment and defines the frame for relative offset decoding. Relative position (u, v) is then
23

---

Published as a conference paper at ICLR 2026
generated in the local frame of the selected map segment. Heading and velocity decoded last, conditioned on the selected geometry and pose to avoid implausible combinations. Without this ordered structure, flat decoding often produces invalid combinations, e.g., a pedestrian way lane or a vehicle with inconsistent orientation and lateral velocity. Our sequential decoding mirrors the causal structure of how agents are realistically introduced into traffic scenes, improv- ing robustness and realism in downstream simulation. This compact, conditioned decoding ensures that agents are initialized in contextually appropriate map segments with semantically valid shapes, poses, and velocities. The output relative state tokens are then concatenated and passed back to the main decoder to form the final <RS> token.
high-
Real-world conversion. The tokenized agent state is decoded into global pose and velocity using the geometry of the selected map segment. Given the segment pose (æ yA, 1/x) and the predicted relative offset (u, v, deb, Ux Uyy in the local frame of the segment, the agent's global state computed as:
x x + u cos 1hx - usin g/x,
(24)
y yı usin + v cos vA,
(25)
V <² + 6ep,
(26)
global = Ur cos د/ا - U/ sin 1/x
(27)
m
global
Ur sin Φ + By cos v/x.
(28)
At inference time, new agent state token groups are generated via autoregressive sampling. the
resulting (z, y) position lies within occupied region causes overlap with existing bounding boxes, the sampled <MS> or <RS> tokens are rejected and resampled up to a fixed number of retries. A maximum of Nnew agents can be injected per step.
For existing agents that persist from the previous timestep, SceneStreamer bypasses the relative state prediction head and instead deterministically generates their agent state token group using their observed global state. Specifically, we first identify the closest map segment A, Then we compute the relative state (u, v, dØ, U2 vyy) by transforming the agent's global pose and velocity into the local frame of A₃. These values are used to obtain the corresponding <RS>> token. The four agent state tokens (<SOA>, <TYPE>, <MS>, and <RS>) can then be constructed directly via embedding lookup and state-forcing. This allows SceneStreamer to seamlessly unify dynamic agent injection (via sampling) and agent motion continuation (via projection), ensuring closed-loop autoregressive simulation across variable-length agent sets.
E TRAINING AND INFERENCE DETAILS
E. DATASET AND PREPROCESSING
Map Preprocessing. We preprocess the vectorized HD map into a fixed-length token represen- tation by segmenting the raw polylines into discrete map segments. Each polyline is split into segments of approximately 10 meters length. To limit memory and computational cost, we cap the total number of segments 3000 per scene. If the number of segments exceeds 3000, we sort all segments by their Euclidean distance to the SDC's current position and retain the closest 3000 segments.
Each segment consists of up to 30 points and is represented by a 27-dimensional feature vector per point. The segment-level position and heading are computed by averaging the position and heading of all points in the segment. The levet-level features include geometric information, heading encoding, and semantic labels derived from MetaDrive map types. Specifically, each point feature contains:
Start and end coordinates: dimensions (I,, Vor Ea. Fes Ve: 26)
Direction vector:
dimensions (dx, dy, dz)
Heading: raw heading, sine, cosine (3 dimensions)
Point length (1 dimension)
24

---

Published as a conference paper at ICLR 2026
Binary map type indicators (12 dimensions): is.lane, is.sidewalk, assroodd.ounddrr.... is.road.line, is.broken.line, is is yellow 1ine, is white 1ine. is driveway, is crosswalk, is speed bump, is stop sign
Segment length (1 dimension)
Valid mask (1 dimension)
Formally, the full feature vector for each map point is a 27-dimensional vector:
f [Zaa Msy Ray Fex Mes Ze, dx, dy, dz, sin( cos(O), 40,),,44) .... (12, m,]
(29) binary indicators for map semantics, is total road length,
where 7 is the segment length, and m is binary mask indicating validity. The processed segments are stored as a tensor of shape [M, 30, 27] accompanied by a binary mask of shape [M,30] for downstream consumption in the Transformer encoder.
Traffic Light Preprocessing. We preprocess traffic light tokens from the raw data by extracting their spatial and semantic states over time and aligning them with the map representation. As discussed in appendix C.2 for each traffic light, we will prepare this information:
the traffic light ID (index),
the map segment index it is attached to,
the traffic light state (semantic),
the position of its stop point (spatial),
and its heading aligned with the associated map segment.
The ground truth prediction for a traffic light token at each timestep its state in the next step, formulated 4-w classification problem (unknown, green, yellow, red).
Agent and Motion Preprocessing. We only select agents that are valid at t = 10, which is desig- nated the "current" step Waymo Open Motion Dataset (WOMD).
Agents are reordered based on type so that vehicles appear first, followed by pedestrians and then cyclists.
To improve training efficiency. we introduce configurable maximum agent count N. If scene contains more than N agents, rank all agents by their cumulative movement distance and retain the top-N most dynamic ones. The remaining agents are masked out at all timesteps, reducing the number tokens processed per scene.
For each agent, we extract the following attributes:
Agent ID (used for dedicated ID embedding),
Agent type (vehicle, pedestrian, or cyclist),
Agent shape (length, width, height) at the current timestep,
Agent position and heading at each timestep (used locate tokens spatially),
A 2D velocity vector in the agent's s local frame.
The motion label in 33 x 33 + = 1090 candidates (one them is (stan)
This information is sufficient for constructing motion tokens described in the model architecture. Agent motion labels are generated following the tokenization scheme in appendix C.2 At the first timestep when agent becomes valid, special start label Asan is used generate its first motion token <MO>. For subsequent steps, the model uses the previous token Pi,r-1 as autoregressive input. The ground truth motion label is defined as the motion label at the next step. We skip loss computation for any motion token if the agent is invalid at the current step or if the next-step label unavailable due the agent becoming invalid the following timestep.
Agent State Ground Truth Preprocessing. Agent state tokens used to autoregressively gen- erate new agents into the scene during scenario generation. These tokens encode where, what, and how to instantiate agent within the current simulation state.
25

---

Published as a conference paper at ICLR 2026
At each sparse timestep (sampled every 5 steps in WOMD), iterate over all valid agents and compute token values and features as follows:
Closest Map ID: For each agent, identify the nearest valid map segment based on Euclidean distance and relative heading difference. Only map features with angular deviation less than 90" considered valid.
Relative Feature Encoding:
The agent's state is expressed as a 8D vector relative the closest
map segment:
2D position offset rotated into the local map frame, heading difference relative to the segment heading, velocity vector rotated into the map frame,
agent shape (length, width, height).
Agent ID: Each agent assigned a unique ID from 0 to 1, where N the number of agents in the scene.
Intra-step Index: We assign each token a unique intra-step index in {O...., x 4++11 to support position embeddings for agent state autoregressive generation.
Agent Type: The semantic category of each agent (e.g., vehicle, pedestrian, cyclist) is included as a discrete token input.
The relative feature of the agents also discretized into 81 uniform bins and serve the input as well as the GT for the Relative State Head. These components are later embedded and combined via additive token fusion described in appendix D to form the final agent state token representation.
E.2 TOKENIZATION HYPERPARAMETERS
All dynamii tokens in SceneStreamer represented discrete entries in their respective vocab- ularies, akin to words in a language model. Each token type has a dedicated tokenization scheme with different vocabulary sizes and resolution bounds.
Traffic Light Token. Traffic light tokens represent the current state of traffic signal. They are selected from fixed vocabulary discrete states:
O: Unknown,
1: Green,
2: Yellow,
3: Red.
Motion Token. Motion tokens discretize the space continuous action commands. Each motion token corresponds to tuple (a,w) where is acceleration and w is yaw rate. Both are quantized into 33 bins linearly spanning their respective ranges:
Acceleration E [-10, 10] m/s?.
Yaw rate [- 2: El
rad/s.
This results a vocabulary of 33 x 33 = 1089 regular motion tokens, plus one special Psan token, yielding total vocabulary size of 1090.
Agent State Token.
Agent state tokens are composed of three tokens:
Agent Type: chosen from categories:
Vehicle,
1: Pedestrian,
2: Cyclist.
Map ID: selected from up to 3000 valid candidate map segments per scene.
Relative State Feature: 8-dimensional vector. Each bin is indexed into shared vocabulary of size per dimension, and all attributes are tokenized independently. The binning bounds
26

---

Published as a conference paper at ICLR 2026
position position
E
[ 10, 10] m
velocity.x
E 0,, 30] m/s
velocity-y
E [-100, 10) m/s
heading
E
21 2 rad
length
E
0.5, 10 m
width
e
0.5, 3] m
height
E 00.5, 4) m
E.3 MODEL TRAINING AND INFERENCE DETAILS
Loss Function. All prediction heads in SceneStreamer are trained using the standard cross- entropy loss. For motion and agent state tokens, loss is only applied to valid entries. Specifically, we exclude tokens if the agent is invalid at the current timestep if the corresponding ground truth (e.g., next- step motion) is undefined.
Training Schedule. SceneStreamer is trained two stages:
Pretraining: We first train a base model using only traffic light and motion tokens. The agent state decoder is disabled during this phase.
Finetuning: We then finetune the pretrained model with the agent state decoder enabled, jointly predicting agent state, traffic light, and motion tokens.
Pretraining runs for 30 epochs with a batch size of 4, while finctuming runs for 5 epochs with a batch size of 1. Early stopping is applied the finetuning phase the minJADE motion metric begins to degrade. We use the AdamW optimizer with learning rate of 0.0003, cosine decay schedule, 2000 warmup steps, no weight decay, and gradient clipping with a max norm of 1.0. During training, we allow maximally 36 (pretraining) or 28 (finetuning) agents in scenario to avoid GPU running out of memory. During inference of course we allow maximally 128 agents.
Hardware. All models trained 8 NVIDIA RTX A6000 GPUs, each with 48 GB of memory. Pretraining takes approximately 5 hours per epoch, while finetuning takes about 12 hours per epoch due to the added complexity and reduced batch size.
Inference. At inference time, we sample motion tokens using nucleus sampling with topp 0.95. For all other token types (e.g., traffic light, agent state), we use softmax sampling.
Model Architecture The encoder consists of 2 layers and the decoder has layers. The model uses a hidden dimension dmodel = 128 with 4 attention heads. The full model has approximately 4.6 million parameters, while the base model (excluding agent state components) has 3.3 million parameters.
E.4 REINFORCEMENT LEARNING SETUP
MetaDrive RL Environment. We use MetaDrive (Li et al., 2022) ScenarioEnv, which supports loading scenario descriptions (SD) generated by ScenarioNet (Li al. 2023). Since SceneStreamer also takes SD as input, it is straightforward to implement a bidirectional converter to integrate Scen- eStreamer outputs into MetaDrive's simulation environment. To enable closed-loop training, implement a pipeline that converts predicted agent states from SceneStreamer into ScenarioNet SD the simulation scenario dynamically. During training,
format. MetaDrive APIs are then used we maintain a buffer that stores the ego agent's past trajectory when it previously encountered the same scenario. This trajectory is embedded into the SD before being passed SceneStreamer. Dur- ing SceneStreamer inference, we state-force the ego agent's 's states and actions, generating scenario that reflects the most recent policy behavior. The resulting SD then sent back MetaDrive for simulation and policy training.
Task Setting. The task is defined as following the trajectory of the self-driving car (SDC) while driving as fast possible and avoiding collisions. In the SD sending to MetaDrive, always overwrite the SDC's trajectory by the original trajectory, thus the reward and route completion are always computed against the GT SDC trajectory.
27

---

Published as a conference paper at ICLR 2026
Observation Space. The RL agent receives the following observation at each timestep:
120-dimensional vector representing lidar-like point clouds within a 50 m radius around the agent. Each value lies in [0. 1] and encodes the normalized distance to the nearest obstacle in specific direction, with added Gaussian noise.
vector summarizing the agent's 's internal state, including steering, heading, velocity, and devia- tion from the reference trajectory.
Navigation guidance the form of 10 future waypoints sampled every m along the reference trajectory, transformed into the agent's coordinate frame.
4. A 12-dimensional vector for detecting boundaries of drivable areas (e.g., solid lines, sidewalks) using similar lidar-based point clouds.
The ultimate observation is 161-dimensional vector. The setting follows the original ScenarioEnv setting (Li et al. 2023).
Action Space The policy end-to-end controller producing continuous two-dimensional action vector a E [|1, 1]2, which is scaled and clipped into throttle/brake force and steering angle commands.
Reward Function
The total reward is composed of four terms:
R = Cj Rdisp + C2l smooth + C3 collisiin + Riem
(30)
Displacement reward: Raisp- de d where d, denotes the longitudinal progress along the reference trajectory in Frenet coordinates.
Smoothness penalty: Pameoth = min 1/2,- - [a[0]]) penalizes sudden steering at high velocity Ut, where a(0) is the steering control.
Collision penalty: Poollision = 2 for collisions with vehicles/humans, and 0.5 for static objects (e.g.. cones, barriers).
Terminal reward: Riem +5 for successful arrival, -5 if the agent ends 2.5 from the reference trajectory.
We 0 1, C2 0.5, and 0g in all experiments.
Termination Conditions and Evaluation
Episodes terminate under the following conditions:
1. The agent deviates >4m from the reference trajectory (out of road).
2. The agent reaches its destination (success).
3. The agent fails to complete the episode within 100 steps (the Waymo scenario typically has 91 steps).
Evaluation Metries: Policies are evaluated on held-out validation set of 100 real-world scenarios from the WOMD validation set. We report:
1. Average Episodic Reward: Total accumulated reward.
2. Episode Success Rate: Fraction of episodes that terminate successfully (i.c., reaching goal with- major violation).
3. Route Completion Rate: Fraction of the predefined route (from GT SDC trajectory) completed per episode.
Off-Road Rate: Fraction of episodes in which the agent deviates off-road.
5. Collision Rate: Fraction of the episodes that have collisions.
6. Average Cost: Combined penalty for collisions and off-road violations.
RL Training. We adopt the TD3 algorithm (Fujimoto et al., 2018 implemented Stable-
Baselines3 (Raffin et al. 202T). The training is performed in continuous control setting using the following hyperparameters:
learning rate: 1 x 10-4
28

---

Published as a conference paper at ICLR 2026
learning.starts: 200 steps
batch.size: 1024
tau: 0.005 (for soft target updates)
ganma: 0.99 (discount factor)
train freq: (update after every step)
gradient_steps: 1 (one gradient update per environment step)
action.noise: None
F FREQUENTLY ASKED QUESTIONS
Q1: Does the number of agents have to be fixed at every step? How can new agents be inserted?
The number of agents in SceneStreamer does not need to remain fixed at every step. Agents can be dynamically added or removed by adjusting the corresponding set of agent state tokens, which serve as the representation of the current traffic participants.
At given time step t, the number of agents is determined by how many agent state tokens present. For example, if there are N agents at step t, then one must reconstruct 4N Agent State (AS) tokens by constructing the agent states and subsequently tokenizing them. In addition, there will be motion tokens at this step.
At the next step t + 1, new agents can be introduced or existing set of agent state tokens accordingly. Suppose we want to add a new agent at f + In this case, after state-forcing the first 4N AS tokens corresponding autoregressively generates four additional tokens to construct the initial state of the new agent. Then, during motion generation, the model will produce (N + 1) motion tokens in a single batch-ow one for each of the existing N agents plus the newly added agent thereby ensuring that the new agent's 's motion seamlessly integrated.
removed by modifying the
the existing N agents, SceneStreamer
During inference, we maintain an incremental list of agent IDs. When new agent is introduced, unique ID assigned it. This ID is embedded into the input tokens, acting a positional embedding that identifies the agent to the model. Because SceneStreamer operates in autore- gressive manner,
straightforward to accommodate four new tokens for the agent's 's state and one additional token for its motion. This design allows the model to flexibly expand or shrink the set of agents at any time step without disrupting the overall generation process.
Q2: How does the model know when to stop inserting new agents?
Just like the language model which has a send.of..squennce. token to indicate wants to stop generation, we also have this <start_of srrrttff.ageetssttteess <endLof.agent.state.
token and token prior and post to the agent state tokens. The model will pro- duce the <end of agent states> after it generates the fourth token of the last agent to indicate the stop of generation. In practice the model does well to produce a reasonable number of agents. In our densification experiment, agents as possible. This can be done by just set the output logit of kend_of_agent .tstsssttssss to -inf. In the scenario generation experiment, following the protocol of Waymo Sim Agent challenge, we will know how many vehicles, pedestrians and cyclists there are in a scenario and we will force the model to generate these agents.
manually set the total number of agents 80 to make as many
Q3: What is state-forcing and will this cause information leak in test time?
In SceneStreamer, "state-forcing" specifically refers to the process of first reconstructing the agent states at current step via the forward kinematics (this can be inferred from the states and the predicted motion previous step) and then tokenizing the new states into agent state tokens. Then we bypass the agent state generation but instead just append those reconstructed agent state tokens into the input sequence. Thus, this completely reasonable setting at inference time, and there is no information leak from ground-truth data. In test time, our model runs autonomously without access to ground-trune truth data.
29

---

Published as a conference paper at ICLR 2026
G QUALITATIVE VISUALIZATIONS
Real-world Map
Ground Truth
Motion Prediction
Figure 8: Qualitative visualizations for motion prediction task.
30

---

Published as a conference paper at ICLR 2026
Ground Truth
Scenario Densification
#2
#3
Mode
Mode #2
Mode
Figure 9; Qualitative results for scenario densification. SceneStreamer injects new agents with alistic behaviors such as jaywalking (Scenario #1 Mode #2, S1M2, S3M1), U-turns (S3M3, S4M2), and queueing (S2M2, S3M1). Common failure cases include overspeeding (S2M1), collisions (S4M2, S4M3), and signal violations (S2M1).
31


Reviews:
[{"review_id": "a7GZ8NsJEi", "reviewer": "Reviewer_PzMo", "summary": "This paper introduces InfGen, a novel generative framework for traffic scenario simulation essential for training and evaluating autonomous driving (AD) systems. InfGen models the entire dynamic driving scene, including traffic light signals, agent states, and motion vectors, as a single, structured sequence of discrete tokens. It employs a unified autoregressive Transformer model to generate these tokens step-by-step, enabling continuous and long-horizon scene rollout.\n\nA key contribution is the unified state and trajectory tokenization, making InfGen the first to use a single autoregressive model to produce both agent initial states and motions as a continuous token sequence, addressing the inflexibility of prior two-stage models. The framework features a novel autoregressive generation scheme for agent states by predicting the agent's type, its anchoring map segment ID, and its relative kinematic state, which improves semantic and physical consistency over flat decoding methods.\n\nThe model's design supports dynamic agent injection and infinite scene generation, overcoming limitations of static initialization and fixed agent populations in existing data-driven simulators. The versatility is demonstrated across multiple tasks including motion prediction, full-scenario generation, scenario densification, and closed-loop simulation. Experiments, particularly the deployment in a Reinforcement Learning (RL) pipeline, show that policies trained in InfGen-generated scenarios achieve superior robustness and generalization compared to log-replay baselines, validating its utility as a high-fidelity simulation environment.", "questions": "1. Comparative Motion Prediction Performance: Could the authors provide a comparison of InfGen-Motion or InfGen-Full's ADE/FDE metrics against a few established state-of-the-art motion prediction models (e.g., MTR++, Scene Transformer, or others from the literature review) on the WOMD validation set? This is crucial to quantify the quality of the trajectory modeling component, independent of the initial state generation.\n\n2. Quantitative Diversity Metrics for Scenario Generation: Please provide quantitative metrics (e.g., a simple measure of agent density or a diversity metric like ADE_k/FDE_k across multiple diverse samples of full scenario generation or scenario densification) to objectively support the claims of high diversity and realism against baselines. How diverse are the generated scenarios beyond motion?\n\n3. Role of Agent ID Embedding in ⟨AS⟩ Tokens: For a newly generated agent, the <SOA> token uses a unique Agent ID embedding, EmbAID(i). Since this agent is newly generated, this ID must be incrementally assigned. How is the sequence of future Agent IDs managed by the Transformer during autoregressive generation, and how is the embedding EmbAID(i) for a new, unseen ID i generated or retrieved, given that the ID is learned from the training data's fixed agent indices?\n\n4. Handling of Maximum Agent Count Discrepancy: The paper mentions limiting agents to N=28/36 during training but allowing up to 128 during inference. Can the authors elaborate on how the model generalizes to a significantly larger number of tokens (nearly 4×36×4≈576 tokens at N=36 vs. 4×128×4≈2048 tokens at N=128) for scenes with high agent density, especially considering the Transformer's quadratic complexity without KNN?\n\n5. Causal Flow between <AS> and <MO> Tokens: The Token Group Attention mechanism (Fig. 6) shows <MO> tokens attending to Current <AS> tokens, reflecting the causal flow of motion depending on the current state. However, during the sequential generation at inference time, the full set of <AS> tokens must be generated before the <MO> tokens are generated in a batch. This causality needs clarification: Is the dependency between the final <AS> tokens of all agents at time t and the <MO> tokens at time t fully respected by the attention mask?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "*Originality: The core idea of framing multi-agent, dynamic scenario generation as a unified next-token prediction task using a single autoregressive model (InfGen) is highly original in this domain. Specifically, the autoregressive generation of agent states (Type, Map ID, Relative State tokens: ⟨SOA,TYPE,MS,RS⟩_t) anchored to map segments is a clever mechanism for achieving physically and semantically consistent agent initialization, which is a major advancement over prior non-causal \"flat\" decoding methods.\n*Quality: The technical execution is robust. The introduction of state-forcing unifies the generation of new agents (via sampling) and the continuation of existing ones (via deterministic update), enabling seamless closed-loop simulation across variable agent sets. The detailed motion tokenization using a discretized control input space, optimized via Average Corner Error (ACE), provides a robust foundation for motion prediction quality.\n*Clarity: The paper is generally well-written and logically structured. Key concepts like the token sequence composition ([<MAP>; (<TL>,<AS>,<MO>)_1 ;…]) , the three main token groups (TL, AS, MO) , and the process of agent state generation are clearly explained and effectively illustrated in Figures 2 and 3/7. The ablations and RL experiments clearly validate the core components.\n*Significance: The results are highly significant for the autonomous driving community. InfGen's ability to produce traffic scenarios that significantly improve the robustness and generalization of downstream RL planners is a compelling validation of its utility as a powerful generative simulation platform. The approach directly addresses the critical limitations of log-replay and two-stage scenario generation, moving closer to truly reactive and diverse closed-loop simulation.", "weaknesses": "1. Limited Motion Prediction Benchmarking: While the core focus is scenario generation, comparing InfGen's motion prediction performance only against its own ablated version (InfGen-Motion vs. InfGen-Full) is insufficient. The motion prediction task (Sec 3.2) is standard, and performance should be compared against state-of-the-art motion prediction baselines on the Waymo Open Motion Dataset (WOMD) to properly contextualize the model's trajectory-modeling capability.\n\n2. Lack of Diversity Metrics for Full Scenario Generation: The paper claims InfGen produces \"diverse\" traffic behaviors but only provides qualitative visualizations (Fig. 4, Fig. 9) and diversity for motion prediction (ADD and FDD). No quantitative metrics (e.g., scene-level diversity, number of distinct agent configurations, or scenario-level rarity metrics) are provided for the full scenario generation task (generating initial states + motions from scratch) or the scenario densification task. This omission makes it hard to objectively compare the diversity of generated scenes against baselines like TrafficGen and UniGen.\n\n3. Scalability Concerns and Trade-offs: The paper notes the high memory demand due to long token sequences for dense scenes and the use of KNN pruning for attention. However, the reported training limitations (max 28-36 agents during training vs. 128 max agents in WOMD)  raise questions about the model's true generalization capacity to high-density scenes. An explicit discussion or experiment on the performance drop or change in behavior when transitioning from low-density training data to high-density inference scenarios would be valuable.\n\n4. Clarity on Relative State Decoding: While the autoregressive decoding of the 8 relative state attributes r_i  is a strong point, the decision to use a tiny Transformer with AdaLN for the Relative State Head  is an architectural detail that could be further justified. Since each of the 8 dimensions is discretized and the sequence is fixed, why is a specialized Transformer (and not a sequence of simpler MLPs conditioned on prior outputs, which might be faster) necessary? This sub-module warrants a more in-depth rationale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rRRyYXEiTx", "reviewer": "Reviewer_JzqF", "summary": "Scenario Generation as Next-Token Group Prediction introduces a unified transformer framework for traffic-scene simulation, where InfGen represents heterogeneous traffic elements—maps, traffic lights, and agent states/motions—as discrete tokens within a single autoregressive sequence. This formulation supports dynamic agent injection, long-horizon closed-loop rollouts, and “infinite” scenario generation, enabling multiple downstream tasks such as motion prediction, scene densification, and RL-based planner training without architectural changes. However, while experiments on WOMD demonstrate reasonable realism and diversity, InfGen’s overall motion-prediction accuracy (e.g., mADE) lags behind top-performing baselines, making its empirical performance somewhat less compelling despite the novel formulation.", "questions": "Q1. InfGen’s overall WOMD performance, especially on the core mADE metric, is noticeably behind UniMM and CAT-K. Could the authors clarify whether this gap stems from model capacity, training configuration, or data resolution differences, and how they envision improving accuracy under the same benchmark settings?\n\nQ2. Since InfGen trains on 2 Hz downsampled data while UniMM uses 10 Hz, can the authors provide evidence or analysis showing how temporal resolution affects model fidelity? Would training InfGen on 10 Hz data close the gap, or is the architecture itself sensitive to long token sequences?\n\nQ3. The paper reports that training at 2 Hz already requires 8 × 48 GB GPUs. Which component contributes most to this computational overhead—token sequence length, dynamic agent injection, or multi-head decoding—and is inference similarly resource-intensive?\n\nQ4. Only one ablation (removing autoregressive decoding) is presented. Could the authors include or discuss experiments isolating other design choices, such as grouped vs. flat attention, relative vs. absolute encoding, or fixed-agent vs. dynamic injection, to verify each contribution’s effectiveness?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "● Unified formulation: Modeling the entire scene as a next-token sequence provides a unified autoregressive framework capturing spatiotemporal dependencies among maps, lights, and agents. This “traffic as language” design enhances long-horizon consistency, supports multiple tasks, and enables seamless dynamic scene evolution.\n● Dynamic agent injection: The model can add or remove agents at different timesteps, breaking from the fixed-agent assumption and better reflecting open-world traffic where participants continuously enter and exit. This improves interactivity and scene diversity, producing more realistic closed-loop simulations.\n● Strong downstream evaluation: The experiments are extensive, and it is commendable that InfGen-generated data are directly used for downstream tasks such as reinforcement learning, demonstrating the framework’s practical value and potential for real-world applications.", "weaknesses": "● Limited performance on core WOMD metrics: Despite its novel formulation, InfGen does not achieve competitive results on the core WOMD leaderboard—particularly on mADE, which is the primary metric of the Waymo Challenge. Its overall scores lag behind recent strong baselines such as UniMM and CAT-K, raising concerns about whether the proposed architectural contributions and dynamic scenario generation truly translate into better motion accuracy or downstream utility. The claimed advantage in supporting downstream tasks may also be less convincing if stronger models like UniMM already achieve superior realism and forecasting fidelity.\n● Temporal downsampling and computational inefficiency: InfGen trains on 2 Hz downsampled data from the original 10 Hz WOMD, whereas strong baselines such as UniMM are trained at full 10 Hz using similar or smaller GPU resources (e.g., 8 × 4090). The 0.5 s frame interval is relatively coarse for autonomous driving, where 0.1 s resolution is crucial to capture fine-grained dynamics and trajectory continuity. This reduction likely limits the model’s ability to learn rich temporal features, contributing to its weaker mADE performance. Moreover, despite the lower temporal resolution, InfGen still requires 8 × 48 GB GPUs, suggesting that its unified tokenized architecture introduces significant computational overhead relative to its output fidelity.\n● Insufficient ablation experiments: While the paper introduces several innovations—grouped causal attention, relative attention bias, autoregressive state decoding, and dynamic agent injection—it lacks ablations to isolate their effects. Table 1 only tests the removal of autoregressive decoding, leaving unclear how other components contribute to performance. Evaluating variants such as fixed-agent vs. dynamic injection or with/without relative attention would better clarify which design choices truly drive InfGen’s gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j5DjQ98ATt", "reviewer": "Reviewer_7cTF", "summary": "The manuscript presents InfGen, a scenario generation framework that can generate realistic, diverse, and adaptive traffic scenarios autoregressively. The main advantage is the ability to simulate the addition and deletion of agents in a closed-loop manner during traffic simulation. But the positioned contribution and the experiments are a bit confusing and limited in the current presentation.", "questions": "1. From the description of the authors and my own understanding, the interaction between the ego agent and the background ones is bidirectional, which means that during decoding phase, the interaction effect should be reflected in a manner like bidirectional attention. But the current design is more like a sequential manner. \n\n2. In Table 1, the authors reported the results after relaxing the evaluation protocol by computing MMD over all agents instead of only vehicles within 50m of the ego vehicle. If the authors think this manner is better, why not compare all comparable baselines under this setting?\n\n3. Based on my knowledge, there have already been a few works focusing on closed-loop simulation, and hence the interaction among agents is not new to capture. The main contribution of this paper, to my understanding, is the ability to simulate the changes of participating agents, like adding agents. But the current experiments neither prove that InfGen can perform better than existing baselines in agent simulation, nor show that InfGen can simulate complicated scenarios with frequent agent changes.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Closed-loop simulation\n2. Unified modeling of the whole scenario", "weaknesses": "1. Confusing positioned contribution and experimental setting\n2. Lack of comprehensive comparison", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3879p3Z2HT", "reviewer": "Reviewer_h7bB", "summary": "This paper presents a framework to employ a single autoregressive model that produces both agent states and their motion trajectories as part of one continuous token sequence over a long horizon. Experiments show the proposed method achieves good performance on Scene Generation and Motion Prediction.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The unified token scheme for unified motion and scenario generation is interesting (though not super novel).\n\nThe proposed method is an interesting way to present the two aspects of traffic simulation, scene generation and motion prediction, in a unified autoregressive architecture. This is an interesting way to scale the full simulation process. However, as will be mentioned in the Weakness section, this idea has been discussed in a prior work.\n\n2. Performance of Scenario Generation and Motion Prediction is good.\n\nThe performance of the proposed method on individual tasks of Scenario Generation and Motion Predictions is strong against prior work.\n\n3. Showing improved RL agent performance with InfGen scenarios.\n\nThe highlight of this paper's result is its ability to serve as a scenario generator for an effective data augmentation tool for closed-loop simulation.", "weaknesses": "1. The main technical contribution is not that novel and misses some important references and discussions.\n\nThe main technical contribution of this paper is a unified framework to \"employ a single autoregressive model that produces both agents' initial states and their motion trajectories\". However, this idea is not new and has already been explored in a prior work already published [1]. Therefore, I think it is necessary to discuss and compare against [1] as it targets the same task and has a very similar idea.\n\nOn another level, this paper claimed that prior work like [2] \"separates static and dynamic phases\", which is not accurate. The method in [2] also achieves both scenario generation and motion prediction with a unified model, during which the model has the full unified context of agent placement and movement information. Although it's a diffusion model instead of an autoregressive model, I also think this paper should discuss its difference with [2], a prior work that unifies the two tasks. Note that [2]  does not do the \"fixed-order\" generation like UniGen.\n\n2. Does not conduct extensive experiments on many important aspects of the proposed model.\n\nAlthough the paper shows that the proposed method achieves strong performance for two tasks, there lack of experiments regarding some of the important capabilities mentioned in the main paper:\n\n- Scene editing.\n- Agent insertion/removal during simulation.\n- Traffic simulation (only in the appendix).\n- Extended long-term simulation (no formal experiment, just two demo runs from the supp video).\n\n\n3. Long-term simulation performance issue from the demo video. \n\nI appreciate the authors presenting the demo video of the scenarios generated by the proposed method. I found the long-term simulation capacity of the proposed method is weak:\n\n- (01:46) The model simulates the ego agent moving forward, but the two agents on the right edge of the scenario cannot exit the scene. These agents drives normally to move out of sight of the ego vehicle, which should be removed by the system. However, the system fails to do that and forces the two vehicles to \"come back\" to the region of interest, leading to the unsafe collision and off-road behaviors of these two vehicles. I think this might due to the fact that the proposed method does not learn when to remove exiting agents; therefore, to maintain the status of these agents, it forces them to \"stay inside\" the region of interest. This would become problematic when the simulation duration is longer than the training data.\n\n- Throughout the whole video, I cannot see any agent being added to the scenario. This would be problematic for scenarios like (02:15) where there are less and less agents in the scenario, making the traffic distribution unrealistic.\n\nI wish the author could discuss the reason behind these issues in a systematic way.\n\n\n4. Minor - Method naming conflict with [1].\n\nThe method in [1] is also named InfGen, which has already been published in ICCV. Given that both works focus on traffic or motion generation in simulation environments, this overlap may confuse future citations and discussions. Since the arXiv version of [1] was publicly available months before this submission, the identical naming may lead to confusion when referencing the two works. I suggest the authors consider renaming their method or adding a distinguishing qualifier (e.g., by emphasizing a unique aspect that they can also do initial scenario generation) to avoid ambiguity and ensure clearer attribution.\n\n[1] Long-term Traffic Simulation with Interleaved Autoregressive Motion and Scenario Generation. ICCV 2025.\n\n[2] SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout. NeruIPS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)