Title: Overcoming Challenges of Long-Horizon Prediction in Driving World Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: World models

autonomous driving

generative models

video generation
Abstract: Existing world models for autonomous driving struggle with long-horizon generation and generalization to challenging scenarios. In this work, we develop a model using simple design choices, and without additional supervision or sensors, such as maps, depth, or multiple cameras.
We show that our model yields state-of-the-art performance, despite having only 469M parameters and being trained on 280h of video data. It particularly stands out in difficult scenarios like turning maneuvers and urban traffic. We test whether discrete token models possibly have advantages over continuous models based on flow matching. To this end, we set up a hybrid tokenizer that is compatible with both approaches and allows for a side-by-side comparison. Our study concludes in favor of the continuous autoregressive model, which is less brittle on individual design choices and more powerful than the model built on discrete tokens. Project page with code, model checkpoints and visualization can be found here: https://lmb-freiburg.github.io/orbis.github.io/


Peer reviews:
comments: SUMMARY:
The paper propose a fair comparison between two modeling paradigms for world models: discrete token-based models and continuous models. To enable this comparison, the authors introduce a novel hybrid tokenizer that can be used with both approaches, ensuring that differences in performance are due to the modeling strategy rather than data representation. 

The experiments show that while both models can handle long rollouts, the continuous model is significantly more robust and achieves state-of-the-art performance, especially in challenging scenarios. Besides, the paper also introduces new benchmarking metrics to better capture the shortcomings of existing models.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper introduce new metrics that better capture real-world performance, especially in difficult cases like turning and urban traffic.
- The proposed continuous model achieves state-of-the-art performance with less data and a smaller model size than many competitors, demonstrating both efficiency and effectiveness.

Weaknesses:
- Will the different architectures used for FM and MGM affect the comparability of their performance? Additional ablation studies would provide more convincing evidence.
- While the hybrid tokenizer and empirical comparison are valuable, the core modeling approaches (autoregressive, flow-matching, MaskGIT) are adaptations of existing techniques.

QUESTIONS:
- More ablation study focusing on the architecture would be useful to fully understand the effectiveness of continuous tokens.
- Additional metrics, such as inference speed, would be helpful for understanding the effectiveness of continuous tokens.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Video generative world models are critical for generation of training and evaluation data particularly for corner cases which are difficult to collect manually. This paper proposes a simple continous/discrete hybrid autoregressive (AR) model.\n- The paper compares continous and discrete AR models and demonstrate that continous model works significantly better.\n- The paper proposes that image quality metrics like FID is limited and adds a new trajectory based metric based on pose sequences generated by a recent CVPR'25 paper VGGT.\n- The paper demonstrates significantly better results compared to state of art models in both FID and trajectory based metrics.\n- The impressive impact of the paper is the simplicity and efficiency of the proposed approach with an order of magnitude smaller model and shorter trainings compared to Nvidia Cosmos model which is trained on 10,000 GPUs for three months.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper explores a continous autoregressive model formulation based on Flow Matching. This is a new application of this idea for world models.\n- The model is highly efficient in terms of training compared to the trend of using very large models for world model trainings. This will have a high impact to future work.\n- Significantly better results are obtained compared to sota methods in terms of FID visual quality and trajectory quality metrics particularly for difficult scenarios like turns.\n\nWeaknesses:\n- It is not clear why the proposed method performs well for turns compared to other methods. Please add analysis or discussion to motivate why this happens.\n- The dynamic agents like vehicles and pedestrians are quite important to simulate as the main goal of these videos is to train/test planning or end to end driving models. The performance of dynamic objects doesnt seem to be good relative to static background based on visual inspection of the qualitative videos in supplementary. It will be good to add discussion on this point.\n- Vista paper already had a proposed trajectory metric, this is not discussed and compared to the new trajectory metric.\n\nQUESTIONS:\nPlease see the questions in weaknesses.\n\nLIMITATIONS:\nLimitations are adequately addressed in the discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the difficulty of long-horizon video prediction in autonomous driving world models & proposes a new method called Orbis. There are 3 main contributions: a) It highlights the shortcomings of current driving world models on extended rollouts (e.g. turns) and introduces an evaluation benchmark with metrics to quantify long-horizon prediction quality. b) It proposes a hybrid discrete–continuous tokenization scheme that enables direct side-by-side comparison of discrete token prediction vs continuous prediction under the same token representation. Two corresponding world models are developed and trained from scratch on raw driving videos. c) Through this comparison, the authors find that the continuous autoregressive model significantly outperforms the discrete one in long-term fidelity, achieving SotA results on challenging driving scenarios (e.g. maintaining correct turning trajectories over 20-second rollouts). Notably, the proposed model uses only front-camera video and 469M parameters trained on 280 hours of data, yet surpasses prior larger models in both quality and rollout length. The paper also demonstrates a simple extension for ego-motion conditioning, allowing limited control over the generated trajectory.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n[Novel Approach and Insights]\n\n1. The paper introduces a hybrid discrete–continuous tokenization strategy to fairly compare two paradigms of world models. This is a novel idea by using a unified token space for both a MaskGIT-based discrete model and a flow-matching continuous model, the study provides an apples-to-apples comparison of discrete vs. continuous generative approaches.\n\n2. The authors identify a critical failure mode of existing driving models (generation for long horizons) and contribute an evaluation benchmark with corresponding metrics to capture this. They measure how closely generated trajectories can match the real trajectory over time.\n\n[Comprehensive Experiments]\n\n3. The experimental evaluation is extensive and thorough. The authors test their models on multiple datasets and scenarios, including a challenging turning-manuevers dataset (nuPlan-turns) and two other public driving benchmarks. Both short-horizon (6s) and long-horizon (20s) performance are reported, showing that the proposed continuous model outperforms all baselines. The paper also evaluates a fine-tuned version with steering inputs to demonstrate controllability, and it conducts ablation studies on the tokenizer design (e.g. testing the impact of adding DINO feature distillation or factorizing tokens).\n\n4. The authors provide insightful analysis of their results, e.g. failure modes of the discrete model & comparisons with other continuous baselines (Vista, GEM, Cosmos, etc.). Overall, the experimental section not only demonstrates superior performance but also provides careful oblation studies that strengthen the conclusions.\n\n[SotA Performance]\n\n5. The proposed Orbis model achieves SotA results in driving video generation, while using fewer parameters and less training data. Orbis is particularly good in difficult scenarios like turning where prior methods often drift or collapse.\n\n[High-Quality Writing]\n\n6. The paper is well-written and easy to follow. The introduction clearly motivates the problem. The methodology has sufficient technical details and the experiments come with solid figures & tables. The authors also discussed limitations of the proposed methods.\n\n# Weaknesses\n\n[Limited Theoretical Insights]\n\n1. The paper does not fully explain why diffusion-based video models struggle with long-horizon predictions. They proposed guesses on factors like pretrained biases or the parallel multi-frame prediction in those models, but no conclusive evidence is provided. This limits the theoretical insight of the work. The results show which approach (continuous AR) works better, but not fully why the other approach fails (aside from the discrete token “content copying” problems that they observed).\n\n[Generation Fidelity Limitations]\n\n2. Despite clear improvements, the fidelity and realism of the generated videos still have some limitations. The authors note that certain fine details are “not yet generated reliably”. For example, important objects like traffic lights and street signs often do not appear or are unclear in the output. Additionally, the behavior of other traffic participants in the generated videos is not always realistic. In practice, missing a red light or having a car break traffic rules in simulation could be problematic if the model were used for planning or training an agent.\n\n[Insufficient Downstream Validation]\n\n3. The scope of the paper is largely restricted to open-loop video prediction, and it does not demonstrate the model’s usefulness in downstream decision-making tasks. While long-horizon visual predictions are interesting on their own, it remains unclear how this world model would integrate into an autonomous driving system beyond simulation. The authors have not yet tested the model on downstream tasks, so we don’t know if the improved long-horizon generation actually leads to better planning or control performance.\n\n[Benchmarking and Baseline Clarifications]\n\n4. The comparison against the discrete methods might not capture the full potential of discrete token world models. The paper’s discrete baseline (Orbis-MG) is a novel implementation based on MaskGIT, since the best-performing discrete model (GAIA-1) is not open-sourced. While the authors did their best to build a strong discrete model, it’s worth noting that GAIA-1 (cited as having no issues with long drives) could not be included for direct comparisons. This leaves some uncertainty: the results show the continuous model is better than all public baselines, but we are not sure how it stands relative to the closed-source GAIA.\n\nQUESTIONS:\nSee the Weakness part above.\n\nLIMITATIONS:\nSee the Weakness part above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new video generative method for driving scenarios. The method allows to generate better (and more complex videos) considering a long horizon (> 5s). Thanks to diverse incremental improvements, the model yields state-of-the-art results despite having only 469M parameters and being trained on 280h of video. However, in my opinion, the results lack quality and resolution, making them impossible to reuse in other tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper results are positive and the application is quite recent and relevant.\n- I appreciate all the video results in the appendix\n- The paper showcases good metrics in terms of FVD and FID (distribution metrics, not quality metrics)\n- The authors show consistent 20s generation for autonomous driving videos.\n- The method is more efficient than other methods in terms of data requirements.\n- The authors discuss limitations and social impact.\n\nWeaknesses\n- No real pixel-wise Blind Video Quality Metrics (VQA) are used.\n- The resolution and quality claims are overstatements. The maximum resolution is 512 ×288, and the quality is poor. The resultant videos could not be used in downstream tasks such as training a detector or segmentation model. What is the underlying purpose of such videos? \n- The model is still heavy, trained over 5 days on 72×A100 GPUs. The cost of generating a video is not discussed, but probably ridiculously high.\n\nQUESTIONS:\n- Please add “Ours” in Fig.2. At a quick read, looks like Orbis is a different method.\n- You mention that “high-resolution model operates at 512 ×288” for current standard that is not high-resolution. Could the model generate at HD or FHD. What are the limitations to generate real high-resolution content.\n- Just to clarify, what are the FPS (frames per second) of the model? Assuming that you generate 20s clips, you are really generating 20 frames? Thus, 1 FPS?\n- The high resolution model is trained for 10 epochs over 5 days on 72×A100 GPUs. Could you provide the loss function plots?\n- Could you generate 1 minute clips? Just for the sake of understanding the limitations of the model and possible hallucinations.\n\nLIMITATIONS:\nThe authors discuss the limitations at the end of the paper. A general limitation is the misconception of “high-resolution”, claiming that 512 ×288 is high-resolution is not accurate.\n\nThe authors showcase how to generate videoclips of up to 20s, yet they do not explore longer horizons, even if the model fails the results are interesting for the community.\n\nThe authors evaluate the quality of the videos using FVD and FID, these metrics are often used by the generative community however, only attend at distributions. No real pixel-wise Video Quality Metrics (VQA) are used.\n\nWhat is the purpose of the model? In the introduction you mention “ Recent driving world models [15, 1, 23] built on video diffusion models [4] have made major strides towards generating detailed content in high definition and at high frame-rate”\nLet me doubt about the resolution and quality (high definition). From the real VQA quality perspective -pixel wise- the quality is very bad. I do not think these videos could be used for training other models, which is the main application of generating synthetic data.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-11"}