Overcoming Challenges of Long-Horizon Prediction in Driving World Models
Authors: Arian Mousakhan
Sudhanshu Mittal
Silvio Galesso
Karim Farid
Thomas Brox
Venue: NeurIPS 2025 poster ()
Keywords: World models
autonomous driving
generative models
video generation
Primary area: deep_learning
TL;DR: We propose a world model for autonomous driving capable of long-term rollouts in challenging scenarios, achieving state-of-the-art-results. The model is trained for next frame prediction on video data.

Abstract: Existing world models for autonomous driving struggle with long-horizon generation and generalization to challenging scenarios. In this work, we develop a model using simple design choices, and without additional supervision or sensors, such as maps, depth, or multiple cameras.
We show that our model yields state-of-the-art performance, despite having only 469M parameters and being trained on 280h of video data. It particularly stands out in difficult scenarios like turning maneuvers and urban traffic. We test whether discrete token models possibly have advantages over continuous models based on flow matching. To this end, we set up a hybrid tokenizer that is compatible with both approaches and allows for a side-by-side comparison. Our study concludes in favor of the continuous autoregressive model, which is less brittle on individual design choices and more powerful than the model built on discrete tokens. Project page with code, model checkpoints and visualization can be found here: https://lmb-freiburg.github.io/orbis.github.io/

Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models
Arian Mousakhan'
Sudhanshu Mittal
Silvio Galesso"
Karim Farid"
Thomas Brox
University of Freiburg, Germany
inousakha,,iitaal mit.ataalllllllsoooo....... faridk, brox) uni uni-freiburg-.. de
Abstract
Existing world models for autonomous driving struggle with long-horizon genera- tion and generalization to challenging scenarios. In this work, develop model using simple design choices, and without additional supervision or sensors, such as maps, depth, multiple cameras. We show that model yields state-of-the-art performance, despite having only 469M parameters and being trained on 280h video data. It particularly stands out in difficult scenarios like turning maneuvers and urban traffic. We test whether discrete token models possibly have advantages over continuous models based on flow matching. To this end, we up hybrid tokenizer that is compatible with both approaches and allows for a side-by-side comparison. Our study concludes in favor of the continuous autoregressive model, which is less brittle on individual design choices and more powerful than the model built on discrete tokens. Project page with code, model checkpoints and visualization can be found here: https://lmb-trebbue wwwiburg.github.biokk github.io/orbi.. github.io ..r.ig...bb.....gub.....
Introduction
Intelligent agents operate in complex environments by simulating plausible future states based on past observations. This capacity for imagination allows them to plan toward long-term goals [2 Humans naturally acquire this ability through passive observation and minimal interactions, enabling them to adapt quickly to new and unseen scenarios. Emphasizing the passive observation component of such world models has become particularly popular for the driving world, since large amounts of data exists for this domain. is attractive to circumvent the manual setup of many perception components by learning the visual representation for decision making via the predictive loss of world model.
25 built video diffusion models 6 have made major strides towards generating detailed content in high definition and at high frame-rates. However, Fig. 2 highlights that these models only work well for few frames, especially in case of maneuvers that require generating new content, such as turning. Our evaluation, conducted on the generated videos well the estimated -rojtaajectoris, reveals substantial limitations in how all public driving world models capture state transitions - the key feature of a world model even though the quality of the generated videos is excellent at first.
Recent driving world models
While realistic video generation can be a valuable product, it not the primary objective of world models. As demonstrated in number of influential works (e.g. World Models [2]. Dreamer [3] Cosmos [.) VJEPA-2 2),, their main purpose is representation learning, planning, and policy learning. The quality of state representations and the accuracy of next-state predictions are therefore paramount, with decoded videos serving primarily as an indicator, e.g. of whether the model can
MMann Contributors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

execute a turn stably. Long- hong-ooizoon prediction serves as a more meaningful measure of how well the model captures state transitions, while generalization to complex scenarios reflects its ability to model diverse real-world dynamics. Existing models such Vista and GEM, though conceived with planning as possible application, often fail in non-trivial yet routine situations. In this work we prioritize building a world model with robust state representation and dynamics that handle such cases effectively.
Consequently, a relevant question is whether world models should rely on continuous-ppace latents or use discrete tokens (similar to LLMs) [5]
10 86 for representing world states. The current trend (continuous) models [61 6 [9]. On the other hand, driving world models based on discrete representations and LLM-liKe objectives seem have the edge in terms of rollout duration [3 BI. Among these, the proprietary GAIA-I 1 model showed no issues with turns and long drives. This observation prompts the question of whether the discrete paradigm is really superior to the other for long-term generation, and whether the continuous space is the reason for the observed shortcomings in the current state of the art.
for visual generation goes towards diffusion-basee
To address these questions, we introduce a hy- brid discrete-continuous tokenizer that is com- patible with both types of modeling approaches to be able to compare the two strategies on the same ground. For the quantized-tokee model we developed frame-wise autoregres- sive model based on MaskGIT [IO), whereas for the continuous-token model we developed an autoregressive model based on flow match- ing [46, [4]. Both models were trained from scratch. We also put effort into optimizing the details of the tokenizer. Indeed we find that many of these details are important for the per- formance of the model acting on quantized to- kens. Surprisingly. these details are of little rel- evance for the continuous modeling approach. Both our models can handle long roll-outs, but the continuous approach yields significantly bet- ter results and sets the state of the art by large margin; see Fig.
Model Comparison: FVD vs Training Data
Vista
Parameter Count (M)
400
250 parmes
5e8 params
350
Costnos
FVD
DrivingWorld*
300
250
10a
10a
Training (log scale, hours)
Figure 1: Comparison of model scale, training data volume, and FVD performance of various approaches on the NuPlan-turns dataset. * *Driving- World is trained on the test dataset nuPlan.
Unlike many prior approaches, our world model is trained using only raw video data without using any extra lowelevel regularization objectives, such as structural consistency or pseudo-depth supervision. All implicit perception is learned directly from the presented videos. This makes the approach more scalable and establishes a strong foundation for the development of more controllable models.
We also demonstrate that our model can be modified easily to allow ego-motion control via adaptive layer normalization [58]. To this end, we evaluate the trajectories produced by our world model also settings, where we propose a set of metrics to evaluate realism
in ego- pangiionccomtool-cnnittiiioo and coverage of the requested trajectories.
To summarize, (1) we highlight shortcomings of contemporary driving world models and propose additional benchmarking metrics to make these shortcomings more explicit. (2) We propose hybrid contruuu- tokenizer that is compatible with both discrete and continuous prediction losses and allows us to compare both modeling approaches side-by-side. (3) On its basis, we compare continuous and discrete prediction losses on fair common ground and find a clear advantage in favor of continuous modeling.
(4) demonstrated in Figure I the resulting model much more economical in terms of training data and model size than existing world models. Using only 280 hours of front camera video data, our 469M parameter flow- now-mitching model Orbis already produces sfate-of-the-an performance on horizoorizon rollouts with realistic and diverse trajectories. It excels particularly in challenging driving scenarios.

---

2 Related Work
World Models. The ability of world models to do real-world simulation can be useful for policy
learning [59 26), sample efficient RL [23 24
79 and representation learning [89] Previous
world models have been limited to gaming 22 23 21 and other simulated environments 12 Recent
breakthroughs in video generative modeling 83 have led to future video prediction models an
essential building block for world models.
use BEV (Bird's-Eye Viee- annotations like depth maps, 3D bounding boxes, road maps to generate new scenarios. DriveDreamer [75] incorporated multi- modal input, such as traffic conditions, text prompts, and driving actions, for future frames and action generation. Many other works [77] 90 [76] extended this idea to multi-view video generation. Some recent works [90 [2]
Multiple driving world models 75
also use LLMs and VLMs [47] for spatial reasoning. Although these models show high quality generation, they rely on heavy external knowledge. Such heavy reliance limits the model's s ability to generalize to new environments. In this work, we train a generalizable world model using unannotated front-camera videos and only fine-tune for ego-motion control.
25 30 66] trained predominantly on raw driving video data have shown the ability to simulate realistic future scenes in unseen environments. DriveGAN [3),, among the first works train on real-world driving data, showed realistic future generation with ego-motion and environment controllability. GAIA-1 (30] further enhanced the quality of future prediction and added controllability through text, in addition to action input. Diffusion- based world models [17 25 fine-tuned reneral-purposs pre-trained video generation models like SVD [6] to produce future video predictions at high resolution and high frame rate. Driving world models Vista 17 and GEM [25] demonstrate high-quality rollouts up to 15 seconds. DrivingWorke [3] further enables longer and more coherent rollouts.
Recent driving world models [36 82
Generative models based on vector-quantized tokens like autoregressive 81 38 and masked genera- tive models [87 48 20) have also demonstrated strong performance in video generation due their strong capability in modeling dynamics and representation learning. For world modeling, Genie 7 and GAIA-1 30 have demonstrated generalized world modeling capabilities with interactive control
and long- long-horizon rollouts respectively. We also observe that quantized driving world model can perform long- horizon generation.
Latent Representation Learning. VAEs 60 and VQVAES 73 15 are foundational autoencoding
techniques for learning latent representations used in training latent world models. VAËs produce the continuous latents, commonly used in diffusion [9] 6 and flow matching [66] 61 33 generative models. VQ-VAE produces discrete (quantized) latent codes for LLM-style autoregressive [5] 37 and masked generative modeling [87 77. Following VQGAN work [[5], ideas such as product quantization [4
residual quantization 4T multi-scale residuals 770 [53] spectral decompo-
sition [16 45
have been introduced for image and video generation. Some works also propose
hybrid tokenizers 669 models.
that unify the tokenizer model for both discrete and continuous generative
3 Challenges Faced by Contemporary Driving World Models
Prior works perform well in straight-road driving scenarios but show significantly higher failure rates when faced with difficult maneuvers. For example, in turning events, as rollouts extend over longer horizons, the content generated by these works tends to run out of distribution, producing blurred frames. The degradation of visual semantics and details causes the ego-vehicle to stop prematurely, as the model cannot recover from poorly generated new context. This can be seen in Figure 2] where Vista [17] comes to a halt before 10 seconds. Most of these limitations can be reflected using video quality metries like FVD [7]] computed on several time windows, as shown in Section (..3
We found that these models also show unrealistic ego-vehicle behaviors, such as lateral sliding or jitter artifacts. This could be due to the strong priors inherited from general- purpossd pre- pre-trained video generation models. Such artifacts can be seen in the trajectories produced by these models, as shown in Fig.
These are not well captured by standard metries like FVD or JEDi, whose reliance on pretrained numan-action or general purpose encoders limits sensitivity to driving dynamics and ego-motion.
3

---

Falimated trajectores
25
56
10s
Context (real)
23
GEM
Digiral
Ours
1.0
Vista
Ours
Figure Limitations of state-of-the-art video generation models on turning events. Left: The trajectories estimated from the generated videos show that previous approaches either stop prematurely or drift into an unnatural path. Right: The quality of the corresponding generated frames degrades over time, the models struggle to generate the scenery. In contrast, our method tracks the original trajectory curvature and speed more closely, and can generate novel content beyond the unseen horizon. Videos for a larger set of randomly sampled context frames are linked in the Appendix.
This calls for more targeted evaluations. To bridge this gap, we propose a distribution-level trajectory- based evaluation, detailed in Section 5.2.2 that directly quantifies realism and coverage of generated driving behavior compared
curated dataset of turning events. We evaluate and compare the generated trajectories for Vista, GEM, and our approach, and find the results to confirm our qualitative observations and show the shortcomings of the existing methods.
4 Compatible Discrete and Continuous Prediction Models
The above shortcomings all appear in conjunction with approaches based video diffusion. This modeling approach could be a potential cause of these methods" failure. To enable a fair comparison between discrete and continuous latent world models, we design hybrid image tokenizer that supports both objectives and allows us to evaluate directly which objective better handles the challenges of honghoorzoon prediction in simple and controlled setting. Our study is conducted using two efficient formulations: flow matching for continuous models and masked generative modeling for discrete models.
4.1 Hybrid Image Tokenizer
Preliminary. Given image I E Hx33 the encoder & produces a latent E(I) R H'''"''d, xd. where d is latent channel dimension. The latent then quantized to the closest codebook entry, resulting in q = Q(x) e H "H''xW'xx xd, using a codebook E R with entries. The tokenizer is trained using the VQGAN 15 objective.
Our design. Building upon recent works 43 85 88 we design custom hybrid tokenizer, suitable for both discrete and continuous predictive video modeling. VQ-VAEs (73) typically optimize latent representation learning for pixel-level reconstruction. Prior works show that these representations typically lack desirable properties such as semantic structure [43 To address these limitations, we adopt factorized token design [3] using separate encoders ε, and E to produce semantic tokens x, and detail tokens Xd, respectively. The former are obtained via additional distillation from DINOv2 (55) as shown in the Figure 3 Each output is quantized independently using separate codebooks C, and Ca. yielding q, C. (x.) and qı Qa(xa). We convert the tokenizer into hybrid model by fine-tuning with a 50% probability of bypassing the VQ bottleneck during training.
This simple modification allows single tokenizer to support both discrete and continuous latent

---

Simllrlty
Hybrid Image Tokenizer
Var Vactor Connicstion palion
Inference
Figgon
Samertt
Concalae tallon
DUMM Er
World Model
Reconetrcted Iruge
Voss trage
Xx-a
Werts Model
World Model
Erooded Conteel Fromes
Spata Ataintoon
Temporra Adartico
Forward
Decodlr
Condiliriing Enbediing
Adel
STDIT Block (AN)
Sumplin denobeng urmakinn
(enellone)
The tokenizer provides two semantic and detail representation. These two representations are concatenated and fed into the image decoder and later the world model. During training the decoder receive continuous or discrete tokens randomly in the fine-tuning phase. World Model: To generate the next frame, the model receives either sampled Gaussian noise fully masked tokens as the target frame, along with encoded context frames. 'hee model progressively denoise or unmask the target frame. This iterative sampling process repeated to generate target frame. Inference Rollout: During inference, the world model autoregressively generates next frame. This process repeats for the desired number frames the rollout sequence.
Figure 3: Image Tokenizer:
representations. The final continuous representation is (x,: (x) and the corresponding quantized representation is qa). The decoder reconstructs the image from the final representation.
4.2 Latent Space World Model
We formulate world model next-frame autoregressive model for both discrete and continuous objectives, as demonstrated in Figure The model receives the context frames Xo:N-1 and a target frame x²_ initialized by noise complete mask. The model predicts the next frame x iteratively over multiple M denoising or unmasking steps. During inference rollouts, the model updates its context by appending the most recently generated frame x/= discarding the earliest context frame Xạ of the previous inference step. This sliding- wwindow process is repeated for each next-frame generation get long-horizon predictions in the latent space (Figure 3 For visualization. each generated latent decoded into an image using the tokenizer decoder.
In this work, we consider the flow matching and the masked generative modeling objective
objective for the continuous world model 46 49
10 87 for the discrete world model.
4.2.1 Flow Matching
We follow the flow matching (FM) objective introduced by Lipman [46]: we define forward trajectory from the data distribution standard normal distribution via linear interpolation:
x
7)x + TC,
[0. L]. c0,
(1)
To use the flow matching objective for next frame prediction, the corrupted target frame x is conditioned on previous frames N1.N-1- The model predicts x Xy:N-1), the velocity that would take x towards the Gaussian prior. We train the model with the following objective as:
[o1. [[lv(x)y: X0 - (e [)
At inference time, we sample noise vector as the new target frame and iteratively transform towards the data manifold. At each iteration the model calculates the velocity conditioned context frames and update the target frame as:
X - 8r v(xXxxxo.-1)
(3)
5

---

where 5, is the step size used to update the target frame at time step After integrating from 7 =
0, the resulting latent x is the generated next frame in latent space.
4.2.2 Masked Generative Model
In the discrete setting, we extend masked generative modeling (MGM), following the MaskGIT objective [0]. from image generation to next-frame prediction. The encoded latents of image frames are represented using discrete tokens q.
During training, we apply binary mask M 10,1 to the target frame qN. resulting in the masked frame q qN o M + [MASK] о (1-M). where [MASK] is a learned special token. The masking ratio for the whole frame is sampled uniformly from 0% to 100% using a predefined scheduler. The MGM model takes as input the concatenated sequence of context frames and the noised target frame (qo:N-:: qV training objective is standard cross-entropy loss, defined as follows:
and is trained to predict all the token IDs of the target frame. The
Σ log pa (ax) 9o:N-I, 1..N-1..%.)
CCR = BM -
where indexes over all token positions in the target frame qv. The model receives discrete token indices from the tokenizer, which discards any pairwise similarity structure among latent tokens. Following [67], to reintroduce this structure, we utilize the similarities between quantized code vectors in the VQ codebooks
extra regularizer improve the training objective. At inference, given a fully masked target frame and the context frames (qu.v-1), the model iteratively predicts and replaces masked tokens. We follow, the confidence- based sampling [IO] heuristics for unmasking the target frame.
4.2.3 Conditioning with Ego-motion
To verify that our model capable of action control, we implement the option for additional condition signals via adaptive layer normalization [58]. We embed steering angle and speed with two-layer MI., and add them to the other condition signals.
Experiments
5.1 Experiment Details
Datasets. To train our world model, we use subsets of videos from the BDDI00K (84) and OpenDV [82) datasets, As shown in Table we select limited number of hours from each dataset and extract frames
Table 1:
Overview of the training
datasets used for the world model.
Hz. In total, we use 280 hours of video data from a combined available total of 2747 hours. For BDD100K, we select the day- clear subset of the training set. From OpenDV we exclude night drives via a brightness filter and uneventful ones by the presence of certain words in the original video titles (see Appendix). We then subsample by selecting evenly spaced 30-second clips. To train the tokenizer we additionally select images from Honda HAD [35), Honda
Total (h)
Used
Frames (M)
Name
(h)
OpenDV
1747
5.67
BDD100K
1000
112
4.02
Total
2747
280
9.69
HDD [63]. ONCE [54], NuScenes [8) and NuPlan 9 to
make the dataset diverse. Our dataset primarily consists of daylight scenarios.
Tokenizer details. For the tokenizer, we employ Transformer-basee encoder and CNN-based decoder. Our tokenizer consists 234 parameters and uses two ViT [1]] encoders initialized with
MAE 27 weights, for the two factorized tokens. To address codebook under-utilizatio issues, we incorporate 1.2-normalized codes 85), low-latent dimension [85] and entropy penalty [8]. For further improvement, we fine-tune the model with implicit regularization as proposed in EQ-VAE 39).

---

Latent world model details. Both continuous and discrete models follow a factorized spatial- temporal (ST) transformer architecture [0]. For high-resolution experiments, we replace the spatial block with Swin (0) transformer block for scalability.
For the FM model, attention mask is applied to the time-attention layers. To improve generalization and frame generation quality, we drop all context frames 50% of the time. When context frames are present, we augment them with noise 50% of the time, similar to prior work [72 we use ODE sampler and take 30 steps. For the MGM model, we also add context noise improve robustness towards context noise, especially for long-rollouts: we replace 10% of the frames and 10% of the overall tokens with a mask token.
DiT' [56] with ST transformer blocks (STDIT), where per-frame causal
18 28 In order to sample the next frame,
Masked generative models often exhibit flickering artifacts caused by inconsistent predictions across the temporal dimension. We train lightweight 30M-parameter temporal refinement module to smoothen spatial flickering artifacts. It is U-Net architecture [65], trained using a flow-matching objective. This module operates purely interfere with the world model. More details are included in the Appendix C.3
post-processing step, on single frames, and does not
Training details. Our higher resolution model operates at 512 x 288 and small-scale model at 256 x 256. Tokenizer compresses the image spatially by 16x. We train latent models with a context of 5 frames sampled at 5Hz. All small-scale models for ablation studies are trained on only the BDD100K subset for one day on 32x A100 GPUs. The higher resolution model is trained for 10 epochs over days 72x A100 GPUs
5.2 Evaluation
5.2.1 Video Generation Quality
We evaluate the quality of the generated videos using FVD 7 and JEDi [52 and we use FID to assess tokenizer reconstruction quality. For comparing with the baselines, we show FVD and JEDi on nuPlan g and Waymo [68] datasets with 800 and 400 samples respectively. To evaluate the models in challenging scenarios, we curate a dedicated validation set of turning events (nuPlan-turna consisting of 400 samples, selected from the nuPlan validation set where the initial yaw rate is at least 0.12 rad/s (~11 std). FVD results on nuPlan and nuPlan-ture are not comparable, since nuPlan is much more diverse dataset compared to the specialized nuPlan-tums dataset. All evaluation datasets constitute unseen testing domains for our model and for the baselines, except for Driving World, which contains nuPlan as part of the training dataset. We choose nuPlan over the similar nuScenes due to the latter's irregular sampling rate, which adds unnecessary confounding factor to the evaluation. Additionally, we evaluate our model using Video Quality Assessment (VQA) metrics including PSNR, SSIM and (78) see Appendix D.2
5.2.2 Trajectory Quality
To evaluate the realism and coverage of generated videos in a manner well suited to driving scenes, where the re- alism of ego motion and driving behavior is critical, we propose distribution-level trajectory-base precision and recall metrics. To this end, we map both real and gener- ated videos to pose sequences using the inverse dynamics model VGGT 774, and evaluate realism and coverage via precision-recal following (40), where the number of nearest neighbors within the distribution determines the distance threshold. To measure distances over the tra- jectory sets, we use discrete Fréchet distance [33 and Average Displacement Error (ADE) [57]. both within the distributions of real and generated trajectories and across
Table 2: Quality of estimated 10s tra- jectories for Vista, GEM, and our model, evaluated on turning events from nuPlan.
Frechet
ADE
Model
Prec.
Rec.
Prec.
Rec.
Vista
0.39
0.45
0.25
0.48
GEM
0.33
0.54
0.27
0.47
Ours
0.47
0.56
0.41
0.51
them (see Appendix [DI for full definitions). The latter is stricter metric, the former is agnostic to velocity differences between trajectories.
We compare the quality of generated trajectories for Vista, GEM, and our approach in (Table 2)
The results show the limitations of existing models in capturing the underlying distribution of ego motion and driving behavior. All approaches perform worse in terms of ADE, indicating difficulty
7

---

Table SOTA results: FVD and JEDi over 6 seconds rollouts 5Hz. Numbers of baselines were computed using their official checkpoints. Lower is better. *DrivigWoord (DW) is trained on the test dataset nuPlan and uses ego-motion control as an extra input. Video samples available in Appendix.
FVD4 Waymo
JEDi Waymo
Model
nuPlan
nuPlan turns
nuPlan
nuPlan turns
Cosmos
291.80
278.19
248.39
0.55
0.31
0.50
Vista 17
323.37
422.58
413.61
0.37
0.44
0.54
GEM 25
431.69
291.84
357.25
0.42
0.35
0.31
DW
298.97
N/A
334.89
1.33
N/A
1.50
Orbis (ours)
134.06
167.57
239.20
0.14
0.33
0.16
in maintaining realistic speeds. Moreover, we achieve the best precision-recal on Fréchet distance, indicating that our predicted trajectories more closely follow the ground-truth paths compared to competing baselines.
5.3 Results
Comparison to SOTA. We compare our method against the state-of-the-ar video world models for autonomous driving: Vista 177, GEM [25), DrivingWorld (DW) [31], and the more general-purpose
Cosmos T in its autoregressive Predict 14B version. We focus our comparison on steering-free unconditional generation, i.e. with previous visual observations sole conditioning, with the exception of Driving World which requires the past trajectory. We use context size of five frames for Vista and DrivingWorld one for GEM and nine for Cosmos per their respective designs. The input control for Driving World is implemented for nuPlan's data format.
Results shown in the Table 3 for prediction horizon of 6 seconds at 5hz. The qualitative results for 20s are shown in Figure 5 (more are included in Appendix). Our method outperforms other driv- ing world models all three benchmarks and both metrics, except for Cosmos on Waymo according the JEDi metric. We further compare results for long-horizon video prediction, shown in Figure J on nuPlan-turns dataset. For each method, FVD com- puted over the entire predicted video in windowed manner, where each window contains 20 frames sam- pled at 5hz. Results show that Orbis based on flow matching outperforms all baselines and maintains sta- ble performance over long-horizon prediction of up to 20 seconds. The discrete counterpart Orbis-MG based on MaskGIT, shows suboptimal performance for shorter horizons but scales well over long hori- zons, surpassing all previous works for the last two windows. As discussed earlier, previous works perform well short horizons but struggle with long-horizon predictions. GEM has higher FVD scores for short-horizon due to its single-frame conditioning design but performs relatively better on long- horizon predictions (more details in the Appendix
2,000
Vista GEM
Coemos
1,500
DrbisMCC (ours)
Orbis (ours)
1,000
500
12
Starting time (5)
Figure 4: Video quality (FVD) over consecu- tive 4s time windows nuPlan-turn. The axis shows the starting time of the evaluated time window.
Ego-motion Control and Evaluation As proof proof-ef-concept for ego ggo-mooion control, we fine-tune a copy of Orbis model for two epochs on 75h of nuPlan videos and IMU data. Following previous literature 17 25 31,, we evaluate the resulting model by computing the ADE [57] between true and generated trajectories, estimated with VGGT. We compare the ADE of the same model with and without steering on 400 5s long nuPlan validation sequences in Table Better trajectory tracking under ego-motion conditioning indicates some degree controllability - setting. Indeed, conditioning capabilities for related models are well documented [56] 4
though in preliminary

---

Table 4: Tokenizer ablation. rFID computed on 10k BDD100k images. FVD on 200 sequences of 60 frames.
Table 5: Ego-motion control: ef- fect on the average error between real and generated trajectories.
DINO
TF
Vocab Size
rFID
FVD +
FVD +
Orbis-MG
Orbis
Model
ADE !
4096
9.33
1331.28
240.34
Unconditional
5.20
X
4096
12.17
1214.34
248.79
ego-motion
2.40
2x4096
9.10
533.28
246.11
Context Frames
26
36
46
56
10s
15c
20s
Figure 5: Qualitative results the Orbis model over 20-sec rollouts (zoom-in for details). Videos and more samples available on the webpage.
Effect of tokenizer design. For the discrete model, adding DINO distillation to the image tokenizer, similar GAIA-1 [30]. leads to lower FVD, shown in Table However, the key factor to enable long-horizon prediction for the discrete model is token factorization. Usage of DINO distillation even leads worse rFID (reconstruction FID). However, token factorization annihilates this difference. Interestingly, while the factorized tokenizer with DINO distillation is very important for the discrete model, the continuous model is robust to these design changes, showing no large change in FVD. These experiments were conducted in the small-scale setting.
Shortcomings of discrete space modeling. Despite being capable of relatively long rollouts, the videos produced by Orbis-MG average stop earlier than its flow matching counterpart, and their duration is very sensitive the sampling heuristics. We investigated this phenomenon and found that at each location, the model's classifier chooses the exact same token as the last context frame approximately 45% of the times (this number 29% for original encoded frames). This is likely because in the discrete space content copying is an obvious and most rewarding choice. While this phenomenon can be mitigated with regularization like context augmentation and a token-similarity based loss, it does not get fully resolved. Additionally, the discrete model fails to capture small motions of objects which crucial for driving scenarios thereby limiting the expressivity of world model.
Consistency over architectures and sampling budgets. To assess the generality of find- ings, we evaluate Orbis FM and MG using three Transformer architectures DiT [56), STDIT (our default), and CDiT (5), and with different sampling budgets. All models
Table 6: FVD scores for Orbis-MG and Orbis-FM model with different architectures. The FVD is over 60 generated frames of BDD val set.
trained for one day on 32 A 110 GPUs in a small-scale set- ting, and FVD scores were computed 200 generated BDD validation videos (60 frames at fps). As shown in T Orbis-FM consis- tently outperforms Orbis-MG MM across all archi- tectures and inference settings (12 and 30 steps). Orbis-FM shows greater consistency across ar- chitectures than Orbis-MG. The architecture shows the best results for both models, with Orbis-FM (ST, 30 steps) achieving the best per- formance overall.
Model
Architecture
12 steps
30 steps
Orbis-MG (discrete)
ST
533.3
571.7
DIT
769.0
981.3
CDIT
1552.3
1718.9
Orbis-FM (conti.)
ST 360.9
246.0
DIT
345.4
274.2
CDiT
410.2
246.1

---

6 Discussion
We investigated an important shortcoming of contemporary driving world models: their struggle with the generation of new content, which makes long roll-outs, turning maneuvers, and realistic trajectories impossible. We introduced an evaluation benchmark and metrics to quantify these problems and tested the hypothesis that modeling in continuous space is the cause of this problem. We found that this is not the case. Based on a side-by-side comparison with a fully compatible hybrid tokenizer, we obtained two driving world models that both provide long roll-outs. However, the continuous model based on flow matching performs much better and sets the new state of the art. The resulting world model has only 469M parameters and was trained only 280 hours of raw video data. This is significantly less than existing models. At the same time, the approach is perfectly scalable. In contrast to many other recent approaches, it only requires raw video data for training. While we were limited on computing resources for scaling the model ourselves, we expect further improvements when scaling the model parameters, the hours of observed data, the image resolution, and the context length.
Limitations: While our investigation showed that world models built in continuous space are advantageous over models built a quantized token space, we were not able to uncover the reason why the much larger public video diffusion models fail on long roll-outs. One of the possible reason for this could be that these models are typically (but not always) derived from pretrained Stable Video Diffusion model. This could introduce biases in the representation, which are problematic for learning relevant state transitions and generating long roll-outs for driving case. We will analyze this in more detail future work.
Apart from this analytic question, our world model has still several limitations, many of which can probably be mitigated by scaling the model along multiple axes. Detailed content, such as traffic lights and street signs, are not yet generated reliably. The traffic actors do not always follow the traffic rules. While our model shows good diversity when running multiple roll-outs with the same context, the generated trajectories do not seem to represent the true probability distribution. While we showed that conditioning modality can be added flexibly to the model, we did not yet investigate the effectiveness of our model on downstream tasks, such as short-term decision making or planning.
In this work, we contributed to the building of world models a technology, which may enable more reliable and cost-efficient autonomous driving and may play a key role in interactive robotics. In its present state, the research is still in its infancy and results that will affect society will still require a few years.
Societal Impact:
Acknowledgement
This work is funded by the German Federal Ministry for Economic Affairs and Energy within the project "NXT GEN AI METHODS" (19A23014R). The authors gratefully acknowledge the Gauss Centre for Supercomputing V. www.gauss-centre..o for funding this project by providing comput- ing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS 34 at Jülich Supercomputing Centre (JSC). The project
accomplished under GCS compute grants- genai-ad and nxtaim. The compute used in this project is also funded by the German Research Foundation (DFG)- 417962828, 539134284, The authors acknowledge support from the state of Baden- Württemberg through bWHPC. We thank Marcel Aach and Sabrina Benassou for their technical support and assistance with the GCS HPC cluster. We thank Rajat Sahay for valuable discussions and feedback during the development of this work.
10

---

References
[1] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.0352 2025.
[2] Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint 7xXi::2506009983 2025.
[3] Zechen Bai, Jianxiong Gao, Ziteng Gao, Pichao Wang, Zheng Zhang, Tong He, and Mike Zheng Shou. Factorized visual tokenization and generation. arXiv preprint arXiv:2411.1661 2024,
[4] Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. arXiv preprint urXiv:24/2.0383 2024.
[5] Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15791-15801, 2025.
[6] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
[7] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
[8] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[9] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong. Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106. 1810, 2021.
[10] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 11315-11325 2022.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 / 1929, 2020.
[12] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference robot learning, pages 1-16. PMLR, 2017.
[13] Thomas Eiter and Heikki Mannila. Computing discrete fréchet distance. Technical Report CD-TR 94/64, Christian Doppler Laboratory for Expert Systems, TU Vienna, 1994.
[14] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[15] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12873 12883, 2021.
[16] Carlos Esteves, Mohammed Suhail, and Ameesh Makadia. Spectral image tokenizer. arXiv preprint 2xXiv22412.09607, 2024.

---

[17] Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang. and Hongyang Li. Vista: generalizable driving world model with high fidelity and versatile controllability. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[18] Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. Adaworld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503.18938, 2025.
[19] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Factorizing text-to-video generation by explicit image conditioning. In European Conference on Computer Vision, pages 205-224. Springer, 2024.
[20] Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martin-Martin, and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206.188% 2022.
[21] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Ad- vances in neural information processing systems, 31, 2018.
[22] David Ha and Jürgen Schmidhuber. World models. arXiv preprint /803. 2018.
[23] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.0160. 2019
2019.
[24] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02199 2020.
[25] Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Pedro Rezende, Yasaman Haghighi, David Brüggemann, Isinsu Katircioglu, Lin Zhang, Xiaoran Chen, Suman Saha, et al. Gem: A generalizable ego-vision multimodal world model for fine fine-grained ego-motion, object dynamics, and scene composition control. arXiv preprint arXiv:2412.111I
2024.
[26] Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. Large-scale actionless video pre-training via discrete diffusion for efficient policy learning. arXiv e-prints, pages 2402, 2024.
[27] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[28] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.1322) 2022.
(29] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503.025333333 2015.
[30] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309. 77080, 2023.
[31] Xiaotao Hu, Wei Yin, Mingkai Jia, Junyuan Deng, Xiaoyang Guo, Qian Zhang, Xiaoxiao Long, and Ping Tan. Drivingworld: Constructingworld model for autonomous driving via video gpt. arXiv preprint arXhv:2412.1950s, 2024.
[32] Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuging Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. Adriver-i: A general world model for autonomous driving. arXiv preprint 2023.
arXiv:2311.133
[33] Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint 1xX1:24100.05944, 2024.
[34] Jülich Supercomputing Centre. JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre. Journal of large- scale research facilities, 7(A188) 2021.
12

---

[35] Jinkyu Kim, Teruhisa Misu, Yi-Ting Chen, Ashish Tawari, and John Canny, Grounding human- to-vehicle advice for self driving vehicles. In The IEEE Conference on Computer Vision and Pultern Recognition (CVPR), 2019.
[36] Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 5820-5829, 2021.
[37] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.141: 14125, 2023.
[38] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang, Hartwig Adam, Ming- Hssaan Yang, Irfan Essa, Huisheng Wang. David A. Ross, Bryan Seybold, and Lu Jiang. Videopoet: a large language model for zero-shot video generation. In Proceedings of the s1st International Conference Machine Learning, ICML'24. JMLR.org, 2024.
[39] Theodoros Kouzelis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. Eq-vae: Equivariance regularized latent space for improved generative image modeling. arXiv preprint arXiv:2502.09509 2025.
[40] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, volume 32, pages 3929-3935 2019.
[41] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523-11532. 2022.
[42] Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2drive: Efficient reinforcement learning by thinking with latent world model for autonomous driving (in carla-v2). In European Conference on Computer Vision, pages 142-158. Springer, 2024.
[43] Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. In The Thirteenth International Conference on Learning Representations, 2025.
[44] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning bird's-eye- s-eee-veiev representation from lidar-camera via spatiotemporal transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[45] Bin Lin, Yunyang Ge, Xinhua Cheng. Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00013/ 2024.
[46] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:22 2022.
[47] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in nearal information processing systems, 36:34892-34966, 2023.
[48] Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yanping Xie, Xiao Han, Juan C Pérez, Ding Liu, Kumara Kahatapitiya, Menglin Jia, et al. Mardini: Masked autoregressive diffusion for video generation at scale. arXiv preprint urXiv:2410.20256 2024.
[49] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03000 2022.
[50] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
13

---

[51] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017.
[52] Ge Ya Luo, Gian Favero, Zhi Hao Luo, Alexia Malicouur-Martinaau. and Christopher Pal. Beyond fvd: Enhanced evaluation metrics for video generation quality, 2024.
[53] Xiaoxiao Ma, Mohan Zhou, Tao Liang, Yalong Bai, Tiejun Zhao, Huaian Chen, and Yi Jin. Star: Scale-wise eet-tto-mmaee generation via auto-regressive representations. arXiv preprint arXiv:2406. /0797, 2024.
[54] Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, al. One million scenes for autonomous driving: Once dataset. arXiv preprint arXiv:2 1037, 2021.
[55] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wei Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[56] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 4195-4205, 2023.
[57] Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In 2009 IEEE 12th international conference on computer vision, pages 261-268 IEEE, 2009.
[58] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[59] AJ Piergiovanni, Alan Wu, and Michael S Ryoo. Learning real-world robot policies by dreaming. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7680-7687. IEEE, 2019.
[60] Lucas Pinheiro Cinelli, Matheus Araújo Marins, Eduardo Antúnio Barros da Silva, and Sérgio Lima Netto. Variational autoencoder. In Variational methods for machine learning with applications deep networks, pages 111-149. Springer, 2021.
[61] Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching- Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024.
[62] Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03066,, 2024.
[63] Vasili Ramanishka, Yi-Ting Chen, Teruhisa Misu, and Kate Saenko. Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[64] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[65] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MIICAA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234 241. Springer, 2015.
[66] Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. Gaia-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523, 2025.
14

---

[67] Nedko Savov, Naser Kazemi, Mohammad Mahdi, Danda Pani Paudel, Xi Wang, and Lue Van Gool. Exploration-driven generative interactive environments. arXiv preprint arXiv:2504.02515, 2025.
[68] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446-2454, 2020.
[69] Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. HART: Efficient visual generation with hybrid autore- gressive transformer. In The Thirleenth International Conference on Learning Representations, 2025.
[70] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839-8488
2024.
[71] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marein Michalski, and Sylvain Gelly. FVD: A new metric for video generation, 2019.
[72] Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408. /4837,2022
[73] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[74] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. arXiv preprint arXiv:2503.1 2025.
[75] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards seal-world-driv world models for autonomous driving. In European Confer- ence on Computer Vision, pages 55-72. Springer, 2024.
[76] Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 14749-14759, 2024.
[77] Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 6902-6912, June 2024.
[78) Haoning Wu, Erli Zhang. Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 201 20144-00154, 2023.
[79] Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-training contextualized world models with wilthe-wild videos for reinforcement learning. Advances in Neural Information Processing Systems, 36:39719-39743 2023.
[80] Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, and Hongkai Xiong. Spatial-tempordl transformer networks for traffic flow forecasting. arXiv preprint arXiv:2001.022 002000 2020.
[81] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint (rXiv:2/04.1015: 20251
[82] Jiazhi Yang. Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng. Ping Luo, Jun Zhang, Andreas Geiger, Yu Qiao, and Hongyang Li. Generalized predictive model for autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
15

---

[83] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong. Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprini AxXi::440.000777 2024.
[84] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3636-2645 2020.
[85] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge. and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:21 10.010..421 2021.
[86] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuse Yang, Yuan Hao, Irfan Essa, et Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459-10469, 2023.
[87] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (0459-10469)
June 2023.
[88] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion-tokeniizer is key to visual generation. arXiv preprint arXiv:2310.05737,222 2023.
[89] Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Xueyang Zhang, Yida Wang, Guan Huang, Xinze Chen, Boyuan Wang. Youyi Zhang, et al. Drivedreamer4d:: World models are effective data machines for 4d driving scene representation. arXiv preprint arXiv:2410. 13571, 2024.
[90] Guosheng Zhao, Xiaofeng Wang. Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xin- gang Wang. Drivedreamer-2: Lim-enhanced world models for diverse driving video generation. arXiv preprint arXiv:2403.06844 2024.
[91] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magievideo: Efficient video generation with latent diffusion models. arXiv preprins arXiv:2211. 10018, 2022.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract summarizes the problem tackled in this paper and our contri- butions. The introduction further explains the motivation of the paper and lists down the contributions and findings of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: We clearly discuss the limitations and open questions of the work Section of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: NA
Justification: All formulation used in the paper are referenced and the empirical perfor- mances reported in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We clarify all details about the experiments, required reproduce the results. Some details are included in the main manuscript and others are included in the Appendix. Additionally, we open- sourced our model and code to the community.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All the datasets used in this work are publicly available. We have open-sourced the model checkpoints and code. Additionally, we provide clear and detailed instructions, required to reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-sourge benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
gewiii/gguee//eeeeeeeioiiiieseee CodeeSumiiiseeeeee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The choice of byperparameters and experiment details are clearly mentioned either in the main script appendix. Full code is available publicly.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: As training of latent world models is quite expensive, we are not able to train each model over multiple random seeds. The models in this work are trained on large datasets and evaluated on multiple evaluation sets.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details of compute resources used for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We Do not violate any code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: At the end of Section 6, we discuss the social impact of our work. We discuss how this work can contribute to building reliable and cost-efficient autonomous driving systems.
20

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: NA
Justification:
The studied topic of research is still in its very early stage of development.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We clearly cite all works, including papers and code repositories, used in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thooe cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We released our code and is well documented such that readers can reproduce the results provided checkpoints get the same results we provided in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
This work does not involve any study or usage of LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htttpp///earriacccc///fofeerrrre for what should or should not be described.
23

---

Appendix
A
Video Rollouts
We include qualitative examples in video form, embedded in the https::/ github
io/oz orooorii..... www.i..gitub...................e hotthuu..o/oph page. We have divided the qualitative examples into sections.
A.1
Comparison with the state-of-the-art
Here we show videos generated by our method beside those generated by the baseline approaches (Vista [17.. GEM [25]. Driving World
World [31]. Cosmos MT), for the same initial condition frames. We include videos generated by our method for both continuous (FM) and discrete (MG) version.
These videos showcase the superiority of our model in dealing with content generation after turning events. Orbis (FM) can generate more realistic scenes and objects than its discrete (MG) counterpart. Moreover, in the fifth scene our model is the only one to halt at a stop sign, generating the passing of a pedestrian and a car.
Along the generated videos, we display the estimated trajectories for Orbis, Vista, and GEM. These show the unrealistic ego-motion that the SVD-based methods produce in some cases. Trajectories estimated using the VGGT model [74].
A.2 Performance in different scenarios
Here we show our videos generated by our model on straight drives, turns, and urban scenes. Moreover, show how our model can generate diverse videos when starting from the same initial condition frames.
A.3 Randomly sampled videos
Here we show randomly sampled videos, all generated by our model, for nuPlan, Waymo, and BDD100K. The first two are out-of-domain, whereas the last is in-domain w.r.t. the model's training data.
Even though our approach can generate videos from out-of-domain condition frames, its rollouts stop more often on nuPlan and Waymo samples, compared to BDDI00K.
B
Dataset Details
B.1
OpenDV
We filter the training videos from OpenDV by brightness and by video title. We discard all videos containing any of the following words in their original title: night, scenic, interstate, nature, desert, park, walking. We then discard all videos with average pixel value below 90 in [O, 255] range, in order to keep consistency with the selected BDD100K subset.
From the resulting 1337 videos then discard the first and last 60 seconds (to avoid text and other overlays) and extract a total of 19398 30-second long clips.
B.2 Validation Sets
Here we describe how we obtained the validation sets used in the paper. We will release the annotation files needed to reproduce the validation sets.
B.2.1 nuPlan
For this benchmark, use the validation set of nuPlan (9) at its original sampling rate of 10Hz. We select the validation samples by ensuring a distance seconds between their starting frames and length of at least 20 seconds worth of real frames available for evaluation.
24

---

The total resulting samples are 5878. Due to the computational cost of generating videos for all approaches we evaluate on the first 800 samples.
B.2.2 nuPlan-turns
For this benchmark, we use the validation set of nuPlan 9 We select the starting frames of the validation samples based on three criteria:
distance of least 3 seconds between consecutive samples,
at least 40 seconds worth of real frames available for evaluation.
initial yaw rate of at least 0.12 rad/s, equivalent to approximately 1 standard deviation.
We evaluate on 400 of the resulting 416 samples.
B.2.3 Waymo
This benchmark is based on the validation set of the Waymo Open Datset 688, at its original sampling rate 10Hz.
We select the validation samples by ensuring distance of 2 seconds between their starting frames and length at least 15 seconds worth real frames available for evaluation. We use 400 of the resulting 406 samples selected with these criteria.
C Model Details
C.1 Latent world model:
Training details
Both continuous and discrete models follow a spatial-temporal Transformer architecture. ST- Transformer blocks [80] have interleaved spatial and temporal attention layers. For high-resolutie experiments, we replace the spatial block with a Swin Transformer [50), leveraging windowed atten- tion for efficiency. Our transformer architecture consists of 24 ST-blocks with a hidden dimensionality of 768, split across attention heads. We train models with context of frames sampled 5Hz, using the Adam W [5]] optimizer with a learning rate of 5 x 10-5
C.2 Flow matching
We modify the DiT [56] a STDiT architecture by decomposing temporal and spatial attention. As shown in Table 7 the STDIT not only achieves better FVD but also the frame quality, measured by FID, degrades more slowly over time.
We compute the standard deviation of the training set's encoded representations and normalize each frame by dividing by this value, ensuring unit variance across inputs (64 This normalization occurs for detail and semantic tokens independently. To improve generalization and frame generation quality, we drop context frames 50% of the time. This number reduces 10% after epochs of training. When context frames are present, we augment them with noise 50% of the time, similar to prior work 72
In order to sample the next frame, we use ODE sampler and take 30 steps [46].
Table 7: Comparison of DIT' and STDIT performance. Metrics are computed over 200 sequences, each consisting of 120 generated frames, using the BDD100K dataset.
Name FVD + FID J. frame 30
FID +. frame 60
FID J. frame 90
FID J. frame 120
DIT 287.03
81.46
91.06
98.45
101.91
STDiT
273.69
77.98
85.53
89.99
89.80
C.3 Masked generative modeling
Here, we explain the extra regularizer which is added to improve the training process of the discrete model.
25

---

Table 8: Overview of the objectives used in three training phases of the hybrid image tokenizer. Ap = 2.0, λEq = 0.25, Àg 0.1.
Total
Trainable
Train Mode
Objectives
# Epochs
Phase-1 1
Full model
discrete-only discrete + cont.
Lree + LvQ + Lper + ADLDINO
12
Phase-2
Full model
Phase-l >XELLE Lree + Lper + *GLGAN
3
Phase-3
Decoder-only
discrete + cont.
5
Since, the model takes discrete token indices input from the tokenizer, it discards any pairwise similarity structure of the latent tokens. To reintroduce this structure, we utilize the similarity matrix E k over the K codebook vectors and let 8; S, be the jth row corresponding to the ground-truth token index i. Formally, letting us E R.K be the model's output logits for target token with index define
eu/
e
p
p
Ejet./T
(5)
where T and T temperature hyperparameters and pt is treated soft-target for the model output. The objective is to minimize the KL-divergence between pi and pt as
HW Σ
LKD T
DKL 11:))
This is similar to knowledge distillation objective 29 by using soft-targets instead of hard one-hot labels and are known to improve data efficiency and generalization. The overall model training objective is C Loe + ACKL.
which aims enrich relational information
We use = 2, T" 0.2 and À = 0.5 for our experiments.
Refinement module. The discrete masked generative model struggles maintain temporal coher- ence across the full spatial extent of each frame. While it captures important temporal connections to keep the motion of objects consistent across frames and often predicts token with correct seman- tic property, it predicts tokens with inconsistent appearance. This is likely due to the limitations of heuristie-based unmasking scheme. These inconsistencies result in flickering artifacts, which degrades the quality of the video. These artifacts negatively impacts FVD performance, especially for long-horizon prediction, where the corrupted predictions reused context. To remedy these artifacts and compare FVD fairly with continuous baselines, we introduce small video refinement model comprising of 30 M parameters. This refinement module only a post- proccssonns unit and does not affect the world model learning. It follows U-Net architecture with 12 3D-convolutional layers and operates in the latent space. takes four predicted frame latents from the world model as input and outputs the refined continuous latents.
It is trained directly on the tokenizer encoder output, where it predicts clean continuous latents from the corrupted quantized tokens from the tokenizer. To simulate noise, 20% of the quantized tokens replaced with randomly picked top-1000 tokens based the similarity matrix. Ground-truth continuous latents from the hybrid image tokenizer serve as training targets. The model is trained with dow-matching objective to denoise corrupted latents. At inference, refinement applied in woiddng-wnndww window manner over 4 frame latents. sliding one frame at time. Only the last predicted frame latent is retained and updated. We use ODE sampler and take only I step. The refined latents are decoded by the tokenizer to produce the final image frame.
a sliding
C.4 Tokenizer training details
We initialize the VIT-Base encoder with pretrained MAE weights. Both encoder branches combined consists of 171.6 6 M parameters. The CNN-based decoder architecture is based on VQGAN [IS] tokenizer and consists of 44.8 M parameters. We use 16-dim latents, each for semantic and detail codebooks. For the final model, train the quantized version of the image tokenizer with codebook
26

---

size of 16384 for each codebook. The model training has three phases. First phase is similar to VQGAN training, but without the adversarial loss [. In the second phase, we fine-tune with scale- equivariance regularization [39]. We only fine-tune the decoder in the third phase with the adversarial loss. Three phases in total comprise of 20 epochs of training. Phase-2 and Phase-3 are trained with a mix of discrete and continuous latents (includes VQ for discrete) to enable corresponding types of world modeling, as shown in Fig. 3 of the main manuscript. In the mixed fine-tuning phases, 50% mini-batches are trained with discrete latents and 50% with continuous latents. Hyperparameters and objective details three phases are included in Table 3 Lyee refers to L1 reconstruction loss, Lper refers to perceptual loss, LEQ refers scale- saal--quuiarriane regularization loss, LGAN refers to the adversarial loss and LvQ refers to the vector- custor-quaniizationn objectives including codebook and commitment losses.
The model trained with a mix of 7 datasets comprising of 2.49 M images. OpenDV dataset accounts for around 90% of the dataset. The split across all datasets included for tokenizer training is shown in Table. 9
We select only daylight images for the dataset.
Table 9: Tokenizer dataset overview.
Name
Frames
OpenDV 82
2.26 M
BDD100K 84
158.6 K
Honda HAD
35
5.1 K
ONCE 54
14 K
Honda HDD
633
5 K
NuScenes 8
3 K
NuPlan 9
47.4 K
Total
2.49 M
D Evaluation Metrics
D.1 Trajectory evaluation metries
To evaluate the distributional fidelity of generated trajectories, error and curve similarity. These metrics serve as distance measures to evaluate distributional fidelity and coverage using precision-recall [40] in the driving trajectory space relevant world model evaluation. Specifically, we represent driving trajectory as sequence of extrinsic transformation matrices T = (T Tr), where each T, comprises rotation (orientation) R, E SO()) and translation (position) Pt E R arranged as T, R2, Pa; 0, 1]. For computing the ADE and Fréchet distances, we consider only the planar positions Pa E R Other parameters within T,, such as the rotation R₁, can additionally be utilized to assess realism aspects like turning behavior and orientation evolution over time.
two primary metries: pointwise
Average Displacement Error (ADE). Given a predicted trajectory + = (PI..... PT)) and ground- truth trajectory t = (p....., Pr), with positions Pt, Ps E IR? the ADE is defined as the average Euclidean distance between corresponding points:
1 (DE(F,T) r) = T Σ -Pll2-
ADE(f,
This metric quantifies pointwise deviations and is sensitive to minor spatial misalignments.
Discrete Fréchet Distance. The discrete Fréchet distance assesses the alignment cost between two trajectories while preserving their temporal ordering:
FD(f,T)
min a.3 s=l...mm
[|Pa())
P3 Il.
where a, are non-decreasing mappings from trajectory indices to points. This metric emphasizes structural similarity and penalizes shape mismatches more robustly than ADE.
27

---

Precision and Recall in Trajectory Embedding Space. To evaluate the distributional alignment between real and generated trajectories, utilize the precision- framework [40]. We first map videos into the trajectory space using an inverse dynamics model (VGGT 74)). Let R [r,) and
G = {g, LM j=1 denote the planar positions trajectories of real and generated trajectories, respectively,
For each real trajectory rş. define the threshold 6f the distance to its k-th nearest neighbor in the real trajectory space R. \ {r:)- Conversely, for each generated trajectory gi, define of the distance its k-th nearest neighbor in the generated trajectory space 9 \ (g) Precision and recall for distance metric d(.; (e.g., Fréchet) are then defined as:
M Precision
1
Σ
1
Br, s.t. d(g).r;) < <0,]]
M j=1
N Σ
Recall
[3g
e g.t. ((r,.g)) <
(8)
N
This adaptive, density-aware thresholding enables reliable evaluation of both fidelity (precision) and coverage (recall), offering realistic reflection of how well the generated trajectories capture the diversity and structure of real-world driving behavior.
D.2 VQA evaluation metrics.
PSNR and SSIM. As shown in the Table we computed average PSNR and SSIM metrics over two shifted video windows of 10 frames (i.e. 2 seconds at 5fps) over 400 generated videos on nuPlan-turns evaluation benchmark. We resize generated videos of all methods to same resolution for a fair comparison. We observe that Orbis model performs marginally better than other methods over the first window. However, all methods converge similarly low numbers in the second window.
Table 10: VQA metries: PSNR and SSIM on windowed video frames over 400 generated
videos on nuPlan-tuna
PSNR
SSIM
Model
frames 0-9
frames 10-19
frames 0-9
frames 10-19
Cosmos
17.29
13.38
0.47
0.38
GEM
14.85
13.73
0.42
0.41
Vista
15.04
13.70
0.44
0.42
DW
17.67
14.70
0.44
0.38
Orbis 18.72
14.75
0.52
0.42
score, which is also used in the data curation/filtering pipeline of Cosmos. In the Table II we report comparison of the DOVER scores on the nuPlan Turns dataset, computed over 17s long videos. We include both the results for the full Cosmos pipeline (including the extra 7B 1B-parameters diffusion refiner, "Cosmos+ref"), and for the pure Cosmos world model with non-generative decoder.
DOVER. We evaluated the models on the DOVER 78
Table 11: Blind VQA DOVER metric on 400 generated videos 17s on nuPlan turns.
Method
DOVERT
Cosmos
19.94
Cosmos+refinement
28.99
GEM
19.76
Vista
21.14
DW
21.92
Orbis
21.34
28

---

D.3 FVD evaluation.
We compute FVD three formats to evaluate both short and nonghorizon predictions. We compute short-horizon prediction over the first 6 seconds of predicted video. Results of short-horizon shown in Table in the main manuscript and Table T2 in the Appendix. Long-horizon FVD is evaluated in two ways: cumulative and chunked. In cumulative FVD evaluation, FVD is computed increasing video lengths starting from 4 seconds, up to 16 seconds. Results for cumulative-FVD on nuPlan-turns dataset are shown in Fig. Fig. the Appendix. Chunked-FVD is computed on consecutive 4 seconds windows taken at different starting timestamps, shown in Fig. in the main manuscript and Fig. in the Appendix on auPlan-turn and nuPlan evaluation sets respectively.
E
More results
700
Vista GEM
Vista
500
600
Coemoe
Coemoo
Orhis (ours)
Orbis (ours)
500
400
FVD
300
200
Video clip durstãon (6)
Starting time [8]
(a) Video quality (FVD) over accumulated 4s time windows on nuPlan-turns. The x axis shows the vidco clip duration in seconds.
(b) Video quality (FVD) over consecutive time windows on standard nuPlan. The axis shows the starting time of the evaluated time window.
Figure 6: (a) Cumulative FVD on nuPlan turns on 400 samples and (b) Chunked FVD on nuPlan standard evaluation set on 800 samples.
Table 12: FVD over 6 seconds at original frame rate of different baseline methods. Lower FVD is better. "Driving World (DW) is trained on the test dataset nuPlan and uses ego-motion control as an extra input.
Model
fps
nuPlan
Waymo
nuPlan turns
Cosmos
10
210.56
249.08
244.80
Vista 17
10
289.95
351.42
353.27
GEM
348.36
218.61
318.73
DW*
31
298.97
N/A
334.89
Orbis (ours)
132.25
180.54
231.88
FVD at original frame rate. Originally, the previously published models were trained and evaluated with different frame rates. The main manuscript evaluated all models at 5hz for a fair comparison, skipping alternative frames if the prediction frame is 10hz. Here, also include FVD scores original prediction frame rates over 6 seconds rollouts, shown in Table 12 The models evaluated at 10hz achieve lower FVD scores than their 5hz counterparts. Despite FVD's sensitivity to frame rate, our model at 5hz still outperforms prior approaches evaluated at higher frame rates.
Cumulative FVD on nuPlan-turns. Figure 6a shows results for cumulative-FVD scores on the nuPlan-turns evaluation set. Our proposed model consistently outperforms other baselines, showing a strong scalable behavior as the prediction window extends.
29

---

Chunked FVD on nuPlan. We also evaluate chunked FVD on nuPlan evaluation set using 800 samples, shown in Fig. 6b Our model consistently outperforms all baseline across all video windows. Cosmos performs relatively well on early prediction windows but degrades very quickly over later windows. In contrast, GEM performs worse in early windows, but extends well for later windows. We observe GEM suffers in the early prediction windows likely due to its single frame context, which causes it to deviate from the ground truth trajectory earlier than other baselines. However, GEM generates better content in later windows, outperforming other baselines over extended predictions.
1,200
w/o refine
w/ refine
1,000
600
Starting time (6)
Effect of refinement module in masked generative modeling (orbis-MG model). Video quality (FVD) over consecutive time windows on nuPlan-turns. The axis shows the starting time of the evaluated time window.
Figure 7:
Effect of refinement module. The refinement module is design to reduce flickering artifacts caused by imprecise decoding of frames in masked generated modeling. We find that refinement module is effective for ann-horizon predictions, where the context is usually corrupted. However, the module has detrimental effect on short-horizon performance. Fig- 7 shows FVD on nuPlan-turns windowed (chunked) evaluation, with and without the usage of refinement module. We observe that the refinement module shows improvement for long-horizon prediction, especially longer than seconds.
Inference speed and memory requirements. Table 13 compares the average times (fps) needed to generate frame, and the required GPU memory. Orbis has the best throughput compared to all other methods. This advantage can be attributed to the smaller size of the Orbis model. By parallelizing Orbis' computation in batches can achieve higher throughput. GEM and Vista are based on the same architecture but use different sampling protocols trading off FPS and VRAM.
We also compare the inference speed of our discrete Orbis-MG model and the continuous Orbis-FM model. We also report the GPU (VRAM) memory requirements for both methods during inference. Orbis-MG shows better inference speed compared to the Orbis-FM model. However, since Orbis-FM achieves significantly better video generation performance, it remains the default choice despite the speed advantage of the discrete model.
Table 13: Comparison of inference speed and VRAM memory requirements of different models.
Method
FPS1
VRAM (GB) 4
Cosmos
0.18
29
Vista
0.58
86
GEM
0.44
45
DW
0.25
10
Orbis-FM (ours)
0.70
24
Orbis-MG (ours)
0.85
21
30


Reviews:
[{"review_id": "i2noxYDJ7O", "reviewer": "Reviewer_KDTT", "summary": "The paper proposes a new video generative method for driving scenarios. The method allows to generate better (and more complex videos) considering a long horizon (> 5s). Thanks to diverse incremental improvements, the model yields state-of-the-art results despite having only 469M parameters and being trained on 280h of video. However, in my opinion, the results lack quality and resolution, making them impossible to reuse in other tasks.", "questions": "- Please add “Ours” in Fig.2. At a quick read, looks like Orbis is a different method.\n- You mention that “high-resolution model operates at 512 ×288” for current standard that is not high-resolution. Could the model generate at HD or FHD. What are the limitations to generate real high-resolution content.\n- Just to clarify, what are the FPS (frames per second) of the model? Assuming that you generate 20s clips, you are really generating 20 frames? Thus, 1 FPS?\n- The high resolution model is trained for 10 epochs over 5 days on 72×A100 GPUs. Could you provide the loss function plots?\n- Could you generate 1 minute clips? Just for the sake of understanding the limitations of the model and possible hallucinations.", "limitations": "The authors discuss the limitations at the end of the paper. A general limitation is the misconception of “high-resolution”, claiming that 512 ×288 is high-resolution is not accurate.\n\nThe authors showcase how to generate videoclips of up to 20s, yet they do not explore longer horizons, even if the model fails the results are interesting for the community.\n\nThe authors evaluate the quality of the videos using FVD and FID, these metrics are often used by the generative community however, only attend at distributions. No real pixel-wise Video Quality Metrics (VQA) are used.\n\nWhat is the purpose of the model? In the introduction you mention “ Recent driving world models [15, 1, 23] built on video diffusion models [4] have made major strides towards generating detailed content in high definition and at high frame-rate”\nLet me doubt about the resolution and quality (high definition). From the real VQA quality perspective -pixel wise- the quality is very bad. I do not think these videos could be used for training other models, which is the main application of generating synthetic data.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper results are positive and the application is quite recent and relevant.\n- I appreciate all the video results in the appendix\n- The paper showcases good metrics in terms of FVD and FID (distribution metrics, not quality metrics)\n- The authors show consistent 20s generation for autonomous driving videos.\n- The method is more efficient than other methods in terms of data requirements.\n- The authors discuss limitations and social impact.\n\nWeaknesses\n- No real pixel-wise Blind Video Quality Metrics (VQA) are used.\n- The resolution and quality claims are overstatements. The maximum resolution is 512 ×288, and the quality is poor. The resultant videos could not be used in downstream tasks such as training a detector or segmentation model. What is the underlying purpose of such videos? \n- The model is still heavy, trained over 5 days on 72×A100 GPUs. The cost of generating a video is not discussed, but probably ridiculously high.", "was_revised": true, "final_justification": "I update my rating to accept. The authors tackled most of the important points of the rebuttal, although some points are open to discussion, the overall paper is solid and a good contribution to the recent trend of world models.", "rebuttal": "Thank you for the positive feedback and thoughtful questions. We address each question below and will incorporate the feedback in the revision.\n\n> **L1 & Q2:** Clarification on high-resolution terminology.\n\n- The term \"high resolution\" in the paper may be misleading, as it is used only to distinguish between the lower resolution (256×256) and the higher resolution (512×288) in our experiments. We will revise this terminology to avoid confusion.\n- The primary limitation for generating HD videos is the computational cost: training at HD resolution would be roughly six times more expensive due to having 6x more tokens, exceeding our current computational budget.\n\n\n> **L4 & W2:** Discussion on the purpose of the driving world model.\n\nWe think generating realistic synthetic data can be a useful application, but it is generally not the primary objective of world models. As demonstrated in a number of influential works (e.g. *World Models* Ha et al. 2018, *Dreamer* Hafner et al. 2019, *Cosmos*, *VJEPA-2* Assran et al. 2025), the main purpose of world models is representation learning, planning, and policy learning. In this context, the quality of the state representation and the prediction of next states distribution is key. The decoded video is only one way to roughly assess to which degree this is successful, e.g. whether the model can successfully complete a turn without becoming unstable and without running into a steady state. The baselines we compare with (Vista, GEM, etc.) are conceived with planning as a possible application, however we observe that they often struggle in non-trivial yet mundane situations. In our work we prioritize building a world model with robust state representation and dynamics, which allows  handling such situations. One could train a more sophisticated high-resolution diffusion decoder post-hoc to improve visual quality, but this is not the scope of this paper. \n\n\n> **W3:** Training compute budget and inference costs for Orbis and other models.\n\n- Orbis training over 5 days on 72 A100 amounts to 9k A100 (40GB) GPU-hours. This is less than all baselines included in this work. For example, Vista was trained for 25k A100 (80GB) GPU-hours, DrivingWorld model for 18k A100-GPU-hours, and the general purpose Cosmos model for more than 20000k H100-GPU-hours.\n- The table compares the average times (fps) needed to generate a frame, and the required GPU memory. Orbis has the best throughput compared to all other methods. This advantage can be attributed to the smaller size of the Orbis model. By parallelizing Orbis' computation in batches we can achieve an higher throughput. GEM and Vista are based on the same architecture but use different sampling protocols trading off FPS and VRAM.\n\n| Method        | FPS ↑ | VRAM (GB) ↓ |\n|---------------|:--------:|:--------------:|\n| Cosmos        | 0.18   | 29           |\n| Vista         | 0.58   | 86           |\n| GEM           | 0.44   | 45           |\n| DW            | 0.25   | 10           |\n| Orbis (ours)  | 0.70   | 24           |\n\n*Table: Inference speed and VRAM requirements for different methods. All numbers are calculated on a H200 GPU.*\n\n> **W1 & L3:**  Pixel-wise video quality metrics.\n\n- We benchmarked Orbis using video quality metrics as requested. As shown in the tables below, we computed average PSNR and SSIM metrics over two shifted video windows of 10 frames (i.e. 2 seconds at 5fps) over 400 generated videos on the nuPlan-turns evaluation benchmark. We resized the generated videos of all methods to the same resolution for a fair comparison. Orbis performs marginally better than other methods over the first window. However, all methods converge to similarly low numbers in the second window.\n\n| Method        | PSNR (frames 0–9) | PSNR (frames 10–19) |\n|---------------|:----------:|:------------:|\n| Cosmos        |   17.29    |    13.38     |\n| GEM           |   14.85    |    13.73     |\n| Vista         |   15.04    |    13.70     |\n| DW            |   17.67    |    14.70     |\n| Orbis         |   18.72    |    14.75     |\n\n*Table: PSNR metrics computed over two shift windows (2 seconds each) from predicted frame 0 to 9 and frame 10 to 19.*\n\n| Method       | SSIM (frames 0–9) | SSIM (frames 10–19) |\n|--------------|:-----------------:|:-------------------:|\n| Cosmos       |       0.47        |        0.38         |\n| GEM          |       0.42        |        0.41         |\n| Vista        |       0.44        |        0.42         |\n| DW           |       0.44        |        0.38         |\n| Orbis        |       0.52        |        0.42         |\n\n*Table: SSIM metrics computed over two shift windows (2 seconds each) from predicted frame 0–9 and frame 10–19.*\n\n- We would like to remark that pixel-wise VQA metrics like PSNR and SSIM are not well-suited for evaluating the quality of generated videos. These metrics rely on exact correspondence between the real and the generated video, which is not sensible, as large parts of the future are non-deterministic and can take different plausible paths, especially on a pixel level.  This is reflected in the tables above, where all methods show similarly low numbers in the second window just after 2 seconds, both for PSNR and SSIM metrics.\n\n- Additionally, the visual fidelity of our model can surely be improved using a more powerful decoder or refiner network can be trained post-hoc, without having to change the world model itself -- similarly to what is done in other works (e.g. Cosmos, GAIA-1).\n   \n\n> **Q5 & L2:** Observations on longer video rollouts.\n\nWe are not allowed to provide longer clips for rebuttal due to the text-only format. In many cases, the model is able to produce longer drives, however, open-ended roll-outs expose two main failure modes: 1) the ego-vehicle stops at a red light, before an obstacle or a stationary vehicle, and does not restart. 2) the model hallucinates a scene change after too rapid shifts in context appearance. Restarting behavior becomes more common when training on more data. We observed this in previous scaling steps. The cause of the hallucinations is likely the limited context size. We agree that this is relevant for the community, and will add it to the discussion. We well also publish longer videos. Moreover, we will release the code and the model upon publication, so everybody can make all experiments they are interested in.\n\n> **Q4:** Orbis model loss function plot.\n\nWe are unable to provide a loss function plot due to the text-only rebuttal format. The training loss curve follows the same trend as  Figure 13 in the DiT paper (Peebles et al. 2022). We can include the loss curve in the camera-ready version.\n\n> **Q1 & Q3:** Other clarifications.\n\n- Our model predicts at 5 fps. Therefore, 20s clips include 100 frames.\n-  In the camera-ready version, we will label our method as \"Ours\" in Fig. 2 to prevent any potential confusion."}, {"review_id": "B3lPEx986z", "reviewer": "Reviewer_DMVr", "summary": "The paper addresses the difficulty of long-horizon video prediction in autonomous driving world models & proposes a new method called Orbis. There are 3 main contributions: a) It highlights the shortcomings of current driving world models on extended rollouts (e.g. turns) and introduces an evaluation benchmark with metrics to quantify long-horizon prediction quality. b) It proposes a hybrid discrete–continuous tokenization scheme that enables direct side-by-side comparison of discrete token prediction vs continuous prediction under the same token representation. Two corresponding world models are developed and trained from scratch on raw driving videos. c) Through this comparison, the authors find that the continuous autoregressive model significantly outperforms the discrete one in long-term fidelity, achieving SotA results on challenging driving scenarios (e.g. maintaining correct turning trajectories over 20-second rollouts). Notably, the proposed model uses only front-camera video and 469M parameters trained on 280 hours of data, yet surpasses prior larger models in both quality and rollout length. The paper also demonstrates a simple extension for ego-motion conditioning, allowing limited control over the generated trajectory.", "questions": "See the Weakness part above.", "limitations": "See the Weakness part above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n[Novel Approach and Insights]\n\n1. The paper introduces a hybrid discrete–continuous tokenization strategy to fairly compare two paradigms of world models. This is a novel idea by using a unified token space for both a MaskGIT-based discrete model and a flow-matching continuous model, the study provides an apples-to-apples comparison of discrete vs. continuous generative approaches.\n\n2. The authors identify a critical failure mode of existing driving models (generation for long horizons) and contribute an evaluation benchmark with corresponding metrics to capture this. They measure how closely generated trajectories can match the real trajectory over time.\n\n[Comprehensive Experiments]\n\n3. The experimental evaluation is extensive and thorough. The authors test their models on multiple datasets and scenarios, including a challenging turning-manuevers dataset (nuPlan-turns) and two other public driving benchmarks. Both short-horizon (6s) and long-horizon (20s) performance are reported, showing that the proposed continuous model outperforms all baselines. The paper also evaluates a fine-tuned version with steering inputs to demonstrate controllability, and it conducts ablation studies on the tokenizer design (e.g. testing the impact of adding DINO feature distillation or factorizing tokens).\n\n4. The authors provide insightful analysis of their results, e.g. failure modes of the discrete model & comparisons with other continuous baselines (Vista, GEM, Cosmos, etc.). Overall, the experimental section not only demonstrates superior performance but also provides careful oblation studies that strengthen the conclusions.\n\n[SotA Performance]\n\n5. The proposed Orbis model achieves SotA results in driving video generation, while using fewer parameters and less training data. Orbis is particularly good in difficult scenarios like turning where prior methods often drift or collapse.\n\n[High-Quality Writing]\n\n6. The paper is well-written and easy to follow. The introduction clearly motivates the problem. The methodology has sufficient technical details and the experiments come with solid figures & tables. The authors also discussed limitations of the proposed methods.\n\n# Weaknesses\n\n[Limited Theoretical Insights]\n\n1. The paper does not fully explain why diffusion-based video models struggle with long-horizon predictions. They proposed guesses on factors like pretrained biases or the parallel multi-frame prediction in those models, but no conclusive evidence is provided. This limits the theoretical insight of the work. The results show which approach (continuous AR) works better, but not fully why the other approach fails (aside from the discrete token “content copying” problems that they observed).\n\n[Generation Fidelity Limitations]\n\n2. Despite clear improvements, the fidelity and realism of the generated videos still have some limitations. The authors note that certain fine details are “not yet generated reliably”. For example, important objects like traffic lights and street signs often do not appear or are unclear in the output. Additionally, the behavior of other traffic participants in the generated videos is not always realistic. In practice, missing a red light or having a car break traffic rules in simulation could be problematic if the model were used for planning or training an agent.\n\n[Insufficient Downstream Validation]\n\n3. The scope of the paper is largely restricted to open-loop video prediction, and it does not demonstrate the model’s usefulness in downstream decision-making tasks. While long-horizon visual predictions are interesting on their own, it remains unclear how this world model would integrate into an autonomous driving system beyond simulation. The authors have not yet tested the model on downstream tasks, so we don’t know if the improved long-horizon generation actually leads to better planning or control performance.\n\n[Benchmarking and Baseline Clarifications]\n\n4. The comparison against the discrete methods might not capture the full potential of discrete token world models. The paper’s discrete baseline (Orbis-MG) is a novel implementation based on MaskGIT, since the best-performing discrete model (GAIA-1) is not open-sourced. While the authors did their best to build a strong discrete model, it’s worth noting that GAIA-1 (cited as having no issues with long drives) could not be included for direct comparisons. This leaves some uncertainty: the results show the continuous model is better than all public baselines, but we are not sure how it stands relative to the closed-source GAIA.", "was_revised": true, "final_justification": "Thanks the authors for partially addressed my questions! After reviewing the rebutall, I decide to keep my original rating of Borderline Accept.", "rebuttal": "Thank you for your positive feedback and thoughtful comments. We provide detailed discussion for all comments and answer each question below. \n\n> **W1:** [Limited Theoretical Insights] Further insights into potential failure factors for other models and future directions. \n-  We speculated in the discussion section about two possible reasons for the failure of previous continuous driving models. Meanwhile, we further investigated the hypothesis that parallel multi-frame prediction would impair long horizon generation, by training our smaller scale model to predict 12 frames at once (i.e. 2.4s ahead, like Vista and GEM). We found our model to work just as well with this setup, ruling out this hypothesis. We will add this result to the paper.\n\n- We could not verify yet the impact of pretraining biases and other architectural design choices due to a limited compute budget (every full training run costs 9k GPU hours). A few additional studies could be conducted in isolation in future work include: 1) comparing the v-parameterization of flow-matching with the $\\epsilon$-parameterization used in of diffusion, as discussed in GameNGen paper [Valevski et al., 2024], and 2) investigating context augmentation strategies, such as injecting noise [Diffusion forcing Chen et al. 2024] or context dropout.\n\n> **W2:** [Generation Fidelity Limitations] Discussion on realism of the generated videos and model's planning ability.\n\n- Fully agreed that we are still some distance away from the point where the world model matches the quality needed for planning and training an agent in real driving scenarios. Our work is just another step in this direction.  \n- We could only experiment at a resolution 288 x 512 due to a limited compute budget. Scaling up the resolution of this model input by $2\\times$ should resolve some of the issues related to missing details. Also increasing the amount of training data would improve the model’s ability to capture more detailed state features. So far, our focus was on improving design choices before solving remaining issues by scaling the model, the resolution, and the data.\n- In this work, our focus is on two key challenges in learning driving world models - long-horizon video prediction and generalization to complex real-world scenarios, such as turning maneuvers and urban traffic. Long-horizon prediction serves as a proxy for evaluating how well the world model learns state transitions, while generalization to complex scenarios shows the model's ability to capture a diverse range of real-world dynamics.\n\n> **W3** [Insufficient Downstream Validation] Discussion on the scope of the paper.\n\nMost existing visual world models which can perform downstream decision-making tasks are limited to restricted or simplistic environments such as Atari, DMLab, or Minecraft. It is feasible to perform closed loop tasks in such settings either due to simpler requirements from the perception module or from the constrained environment setting. So far none of the current real video-based driving world models (including ours) are ready yet to be used in a closed-loop setting. In our work we focus on building a driving world model with robust state dynamics, which allows handling of difficult situations like turning maneuvers or urban driving. We observe that prior methods often struggle in such non-trivial yet common situations in an open-loop setting. We argue that developing models with stable and realistic transitions is critical before they can be effectively trained for planning and decision making. Our additional metrics beyond short-term FVD are early pointers, which we think are relevant for indicating the readiness, such as the degradation rate with increasing roll-out length and diverse plausible trajectories derived from the predicted video.\n\n> **W4:** [Benchmarking and Baseline Clarifications] Comparison with closed-source GAIA-1 model\n- GAIA-1 is not only closed-source; it is not even possible to run the model on new data. All evidence we have is from their paper and the few published videos. Thus, all discussion becomes quite speculative.\n- GAIA-1 having a large number of parameters (6.5B) may provide an advantage in enabling long rollouts. Additionally, they used a 2.6B-parameter decoder to reduce flickering.\n- Although it is not explicitly stated, the shift to a flow-matching objective in GAIA-2 may indicate the advantage of a continuous latent space over a discrete one for the GAIA model."}, {"review_id": "D5ht6RJGck", "reviewer": "Reviewer_hx5U", "summary": "- Video generative world models are critical for generation of training and evaluation data particularly for corner cases which are difficult to collect manually. This paper proposes a simple continous/discrete hybrid autoregressive (AR) model.\n- The paper compares continous and discrete AR models and demonstrate that continous model works significantly better.\n- The paper proposes that image quality metrics like FID is limited and adds a new trajectory based metric based on pose sequences generated by a recent CVPR'25 paper VGGT.\n- The paper demonstrates significantly better results compared to state of art models in both FID and trajectory based metrics.\n- The impressive impact of the paper is the simplicity and efficiency of the proposed approach with an order of magnitude smaller model and shorter trainings compared to Nvidia Cosmos model which is trained on 10,000 GPUs for three months.", "questions": "Please see the questions in weaknesses.", "limitations": "Limitations are adequately addressed in the discussion section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper explores a continous autoregressive model formulation based on Flow Matching. This is a new application of this idea for world models.\n- The model is highly efficient in terms of training compared to the trend of using very large models for world model trainings. This will have a high impact to future work.\n- Significantly better results are obtained compared to sota methods in terms of FID visual quality and trajectory quality metrics particularly for difficult scenarios like turns.\n\nWeaknesses:\n- It is not clear why the proposed method performs well for turns compared to other methods. Please add analysis or discussion to motivate why this happens.\n- The dynamic agents like vehicles and pedestrians are quite important to simulate as the main goal of these videos is to train/test planning or end to end driving models. The performance of dynamic objects doesnt seem to be good relative to static background based on visual inspection of the qualitative videos in supplementary. It will be good to add discussion on this point.\n- Vista paper already had a proposed trajectory metric, this is not discussed and compared to the new trajectory metric.", "was_revised": true, "final_justification": "My comments were suitably addressed and will keep the original rating.", "rebuttal": "Thank you for your positive feedback and thoughtful questions and comments.  We address each question and provide detailed discussions below.\n\n> **W1:** It is not clear why the proposed method performs well for turns compared to other methods. Please add analysis or discussion to motivate why this happens.\n\n- We speculated in the discussion section about two possible reasons for the failure of previous continuous driving models. We further investigated the hypothesis that parallel multi-frame prediction would impair long horizon generation, by training our smaller scale model to predict 12 frames at once (i.e. 2.4s ahead, like Vista and GEM). We found our model to work just as well with this setup, ruling out the hypothesis.\n\n- We could not verify the impact of pretraining biases due to involved heavy compute cost. However, we speculate that prior driving world models such as GEM and Vista, which initialize using general-purpose Stable Video Diffusion model, tend to inherit biases from pretraining data lacking scenarios like sharp turns that are critical for driving tasks. We also observe that the Cosmos model tends to produce smooth, drone-like trajectories, which may similarly stem from biases in the training data. \n\n- Additionally, we believe that it is not a single factor that boosts our model's performance. For instance, training with a flow-matching objective rather than a diffusion objective, such as the EDM used in Vista, can be advantageous. For example, previous works (e.g. *Diffusion Autoencoders are Scalable Image Tokenizers*, *Cascaded Diffusion Models for High Fidelity Image Generation'*) show that v-parameterization (used in flow matching) is more robust than $\\epsilon$-parameterization of diffusion: the most related is \"GameNGen\" (Valevski et al., 2024), where one focus is long-horizon roll-outs in gaming environments. We also incorporated techniques such as adding noise to the context, GameNGen (Valevski et al., 2024) and dropping the context. We observed adding noise to the context improves FVD from 280 to 246 in a small-scale setting.\n\n- The exact reasons stay speculative, since direct comparisons that remove a single potential cause require training an entire new model, which costs 9k GPU hours. \n\n> **W2:** Suboptimal visual quality of dynamic objects relative to static background.\n\n-  We acknowledge that the current quality of the generated dynamics objects is suboptimal based on visual inspection.\n    However, we emphasize that the dynamic agents tend to appear in plausible places at a plausible time and with a realistic speed. This suggests that their internal representation seems to be better than their generated visual appearance suggests. We believe that seeing more of these agents in the data and concurrently scaling the model is necessary to achieve higher quality. Another possible direction could be training a larger diffusion decoder post-hoc (i.e., independent of the predictive model) to enhance visual fidelity. However, even such a decoder would need to see more of these dynamic agents during training to be effective.\n\n> **W3:** Vista paper already had a proposed trajectory metric, this is not discussed and compared to the new trajectory metric.\n-  The trajectory metric proposed in Vista measures the control effect of action conditioning. We will give proper credit to this. However, our proposed trajectory metric operates at a distribution-level, evaluating the quality and realism of ego-motion in the generated videos without action conditioning (Table 3 in paper).\n\n- We also evaluate the effect of conditioning on trajectories (Table 5 in paper) with the trajectory difference metric proposed by the authors of Vista, but we use a state-of-the-art off-the-shelf-model for inverse dynamics (VGGT, Wang et al. CVPR-25). This is consistent with the procedure used in GEM, where the SOTA DroidSLAM was used for inverse dynamics. We refer to both Vista and GEM while using this metric for evaluating action conditioning, in Line 296."}, {"review_id": "jC6rn6dsXA", "reviewer": "Reviewer_NJY5", "summary": "The paper propose a fair comparison between two modeling paradigms for world models: discrete token-based models and continuous models. To enable this comparison, the authors introduce a novel hybrid tokenizer that can be used with both approaches, ensuring that differences in performance are due to the modeling strategy rather than data representation. \n\nThe experiments show that while both models can handle long rollouts, the continuous model is significantly more robust and achieves state-of-the-art performance, especially in challenging scenarios. Besides, the paper also introduces new benchmarking metrics to better capture the shortcomings of existing models.", "questions": "- More ablation study focusing on the architecture would be useful to fully understand the effectiveness of continuous tokens.\n- Additional metrics, such as inference speed, would be helpful for understanding the effectiveness of continuous tokens.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper introduce new metrics that better capture real-world performance, especially in difficult cases like turning and urban traffic.\n- The proposed continuous model achieves state-of-the-art performance with less data and a smaller model size than many competitors, demonstrating both efficiency and effectiveness.\n\nWeaknesses:\n- Will the different architectures used for FM and MGM affect the comparability of their performance? Additional ablation studies would provide more convincing evidence.\n- While the hybrid tokenizer and empirical comparison are valuable, the core modeling approaches (autoregressive, flow-matching, MaskGIT) are adaptations of existing techniques.", "was_revised": true, "final_justification": "While the rebuttal have addressed most of my concerns, some concerns remain — particularly that the core modeling approaches are adaptations of existing methods with limited novelty. Given this, I am maintaining my original score.", "rebuttal": "Thank you for the positive feedback and thoughtful questions. We provide answers to each question below. \n\n> **Q1:** Ablation study on the performance of FM and MGM model with different architectures.\n\nWe evaluated three Transformer architecture variants: DiT [Peebles et al. 2022 ICCV 2023], STDiT (our default), and CDiT [NWM Bar et al., CVPR 2025] - with both the FM and MGM models. All models were trained in a small-scale setting for one day on 32 A100 GPUs and we report FVD scores.\nAs shown in the table below, Orbis-FM model consistently outperforms Orbis-MG across all three architectures in terms of FVD. We evaluated both methods using 12 and 30 inference denoising (unmasking for MGM) steps.\nThe Orbis-FM model also shows relatively stable performance across different architectures, whereas Orbis-MG shows larger variance. \nST architecture gives the best performance for both model types across different architectures.\nOverall, Orbis-FM with ST architecture achieves the best performance with 30 steps, as used in the paper. \nFVD scores are computed over 200 generated videos on the BDD validation set, each over 60 frame rollouts at 5 fps. \nWe will include this ablation table in the revised version of the paper.\n| Model      | Architecture | 12 Steps | 30 Steps |\n|------------|:--------------:|:--------:|:--------:|\n| Orbis-MG   | ST           |  533.3   | 571.7   |\n|            | DiT          |   769.0    |   981.3    |\n|            | CDiT         | 1552.3   | 1718.9  |\n| Orbis-FM | ST         |  360.9   |  246.0   |\n|            | DiT          |  345.4   |  274.2   |\n|            | CDiT         |  410.2   |  246.1   |\n\n*Table: FVD scores for small-scale versions of Orbis-MG (discrete) and Orbis-FM (continuous) with different architectures. The FVD is calculated over 60 generated frames on 200 samples of BDD validation set.*\n\n> **Q2** Additional metrics, such as inference speed, would be helpful for understanding the effectiveness of continuous tokens.\n\nAs shown in the table below, we compare the inference speed of our discrete Orbis-MG model and the continuous Orbis-FM model. We also report the GPU (VRAM) memory requirements for both methods during inference. Orbis-MG shows better inference speed compared to the Orbis-FM model. However, since Orbis-FM achieves significantly better video generation performance, it remains the default choice despite the speed advantage of the discrete model.\n\n| Method     | FPS ↑ | VRAM (GB) ↓ |\n|------------|:-----:|:-----------:|\n| Orbis-MG   | 0.85  |     21      |\n| Orbis-FM   | 0.70  |     24      |\n\n*Table: Inference speed and VRAM requirements of the Orbis-MG and Orbis-FM models.*"}]

Author rebuttal:

Meta-review: ### Summary and Recommendation Justification

This paper presents an important advancement in driving world models, focusing on the critical and previously unsolved challenge of stable, long-horizon video prediction. The authors propose a continuous autoregressive world model based on flow matching, which demonstrates state-of-the-art performance in generating coherent and realistic driving scenarios for extended durations (20+ seconds), particularly in difficult situations like turning maneuvers. A key contribution of the work is a novel hybrid tokenizer that enables a rigorous, direct comparison between continuous and discrete generative approaches, ultimately providing strong evidence in favor of the continuous paradigm. The work is supported by extensive experiments, introduces new relevant evaluation metrics, and impressively achieves its results with a model that is significantly smaller and trained on less data than many of its competitors.

After a thorough and constructive discussion period, reviewers reached a clear consensus for acceptance. Reviewers consistently praised the work for its novel application of flow matching to world models, its impressive empirical results, and its high efficiency compared to the state of the art. The paper makes a well-supported, and timely contribution to a very active area of research.

### Detailed Justification

**Strengths:**

1.  **State-of-the-Art Long-Horizon Generation:** The paper's primary empirical contribution is its demonstration of robust, long-horizon video generation in complex driving scenarios. This directly addresses a well-known failure mode of previous world models, which often degrade or collapse over shorter timeframes. As noted by multiple reviewers, the ability to maintain coherence and realistic ego-motion over 20+ seconds is a substantial step forward for the field.

2.  **Novel and Rigorous Comparative Methodology:** A standout contribution is the design of a hybrid tokenizer that facilitates a fair comparison between continuous (flow-matching) and discrete (MaskGIT-based) generative models within the same token space. This is a valuable scientific contribution that provides clear evidence for the advantange of the continuous approach in this domain, a finding that is of high interest to the generative modeling community.

3.  **Efficiency and Simplicity:** In a field increasingly dominated by massive models trained on vast computational resources, this work is a compelling demonstration of efficiency. As highlighted by Reviewer hx5U, the proposed model achieves comparable results with an order of magnitude less compute and a much smaller parameter count than leading industrial models like Cosmos. This makes the approach more accessible and provides a strong signal about the importance of architectural and objective function choices over sheer scale.

### Addressing Initial Concerns

The initial reviews raised valid concerns primarily related to the evaluation of video quality, the precise purpose of the generated videos, and the lack of certain experimental details. The authors' rebuttal was effective in addressing these points:

*   **Video Quality and Purpose:** In response to Reviewer KDTT's concerns, the authors clarified that the primary purpose of a world model is representation learning and planning, not necessarily pixel-perfect synthetic data generation. Critically, they also conducted **new experiments** during the rebuttal, adding several VQA metrics (PSNR, SSIM, and the blind DOVER score) to their evaluation, which provided a more nuanced picture of video quality and showed their model to be competitive.
*   **Computational Cost and Inference Speed:** The authors provided a detailed breakdown of training and inference costs, demonstrating that their model is not only more data-efficient but also faster at inference time compared to all baselines.
*   **Architectural Ablations:** To address concerns from Reviewer NJY5 about the fairness of the discrete vs. continuous comparison, the authors ran new ablations across three different Transformer architectures, showing that the continuous model's superiority was consistent and not an artifact of a single architectural choice.

The authors' willingness and ability to conduct significant new experiments during the rebuttal period substantially strengthened the paper. The reviewers were largely satisfied with the responses. While some open questions remain, such as a deeper theoretical explanation for the failure of other models and the potential for downstream planning applications, these are seen as avenues for future work rather than critical flaws in the current contribution.

In short, this paper presents a well-executed study with clear and impactful results. It advances the state of the art in a challenging domain, offers a novel and valuable methodological comparison, and does so with a remarkably efficient model.
Decision: Accept (poster)