Title: ReSim: Reliable World Simulation for Autonomous Driving
Venue: NeurIPS 2025
Accepted: yes
Keywords: Autonomous Driving

World Model
Abstract: How can we reliably simulate future driving scenarios under a wide range of ego driving behaviors? Recent driving world models, developed exclusively on real-world driving data composed mainly of safe expert trajectories, struggle to follow hazardous or non-expert behaviors, which are rare in such data. This limitation restricts their applicability to tasks such as policy evaluation. In this work, we address this challenge by enriching real-world human demonstrations with diverse non-expert data collected from a driving simulator (e.g., CARLA), and building a controllable world model trained on this heterogeneous corpus. Starting with a video generator featuring diffusion transformer architecture, we devise several strategies to effectively integrate conditioning signals and improve prediction controllability and fidelity. The resulting model, ReSim, enables Reliable Simulation of diverse open-world driving scenarios under various actions, including hazardous non-expert ones. To close the gap between high-fidelity simulation and applications that require reward signals to judge different actions, we introduce a Video2Reward module that estimates reward from ReSim’s simulated future. Our ReSim paradigm achieves up to 44% higher visual fidelity, improves controllability for both expert and non-expert actions by over 50%, and boosts planning and policy selection performance on NAVSIM by 2% and 25%, respectively.


Peer reviews:
comments: SUMMARY:
This work aims to address the lack of non-expert data in training the driving world model. It collects non-expert data from the driving simulator CARLA. Combining such data with real-world driving videos and simulated expert driving data, the authors trained a controllable world model, ReSim, which reliably simulates high-fidelity future outcomes. Experiments in real-world scenarios exhibit the advancements of the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths:
1. I think the lack of non-expert data in training driving world models is a real problem and collecting them through CARLA is also an efficient way. Meanwhile, the heterogeneous data compilation provides diverse and reliable data for training.
2. The proposed unbalance noise sampling and dynamic consistency loss are inspiring and validated through ablation studies.
3. The supplementary material includes multiple demo videos.

Weakness:
1. The ablation study includes the effect of simulated data. However, as one of the main contributions of this work is the introduction of non-expert data included in training, I wonder what this part alone contributes to the final performance?
2. I don't see an obvious novelty in the model design. However, I think the contribution to the non-expert driving data is good enough.

QUESTIONS:
1. As this paper adapts CogVideoX, do other baselines also use the same video model?
2. Following weakness 1, what are the contributions of expert and non-expert simulation data?

LIMITATIONS:
See weakness and questions. No negative societal impact.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach of learning driving behavior from synthesizing diverse world driving models combining real-world datasets and synthetic data. Their approach is built on a diffusion transformer model, by conditioning on frame latent features, high-level text command and action given by a set of future waypoints. They provide regularization based on dynamic consistency, bias sampling towards higher-noise steps and apply curriculur learning to train the model. For the reward to assess predicted futures, they turn towards CARLA using a infraction score to drive the model towards the best learned policy. They show that this paradigm achieves greater visual fidelity and controllability for expert and non-expert actions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths - \n- Combining real-world data and simulation data for diversity in driving behaviors\n- Performing Video2Reward using Carla infraction score makes the evaluation criteria homogeneous between real and sim\n- Showcases controllable simulator and provides prediction fidelity \n- It performs planning without accessing extra annotation\nWeakness - \n- There is a large amount of domain shift whenever real and simulated data is mixed. However, the authors do not mention how they tackle the problem other than a single sentence mention (line 96).\n- They provide the ratio of samples of OpenDV (4M), NAVSIM (85K) and CARLA (88K). Why are these ratios chosen - what happens when the ratios are switched?\n- If the same scenario (single lane driving) has two different actions, will the algorithm be biased towards the one it has seen more of (weighted more)? Is there a way the authors use towards debiasing the inputs?\n- How much noise can the algorithm tolerate during training and inference?\n\nQUESTIONS:\nWritten in weakness section\n\nLIMITATIONS:\nWritten in limitations section", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes ReSim, a novel diffucion transformer world model method for self-driving that integrates expert real world driving data and simulation data for extreme edge cases not observed often enough in real data. The authors also proposed modifying noise sampling in diffusion, and incorporating an additional latent dynamics loss. The resulting model then can be more reliable in generating outcomes of suboptimal actions, making it excel at trajectory prediciton, planning, and visual simulation.

STRENGTHS AND WEAKNESSES:
###### Strengths
- The paper is clearly written, and was overall a pleasant read;
- I like how this method is able to transfer the knowledge from Carla simulator and enable improved performance on real-world data such as Waymo and nuScenes.
- The experiments are thorough and showcase the strength of the method. I particularly appreciate the conducted human study.

###### Weaknesses
- I found the ablation study underwhelming, with noise sampling and dynamics consistency ablations only showing the qualitative results.

QUESTIONS:
- Q1: How do you pick the evaluation trajectories for human study?
- Q2: Could you provide quantitative results of ablations, as opposed to just image generations?
- Q3: Could you explain in more detail how you collect exploratory data in carla? You say that you have a pre-defined set of action patterns you can execute, what do they correspond to? Do you just let the PDM-Lite execute for a few steps, then interrupt and execute random actions? Or is there something else going on?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to make world model predictions more physically consistent with the given action input. The contributions appear to be training on CARLA sim data with randomized driving signals, as well as a motion consistency loss and a novel noise sampling technique for the underlying diffusion model. The method achieves strong results on benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper addresses an important problem with physical consistency of world models\n- The paper is well written, I greatly enjoyed reading it.\n- Seems to have strong results on benchmarks (e.g. NAVSIM)\n\nWeaknesses:\n- Training on sim data with randomization seems rather obvious and also with possible obvious downsides that could have been explored better. If you deviate from the nominal state of real/expert driving data and benefit form this sim training, do you also visibly see the scene turning more into something that looks like CARLA?\n- Some slight overclaims in the writing: E.g. \"we are the first to demonstrate that integrating both sources overcomes the shortage of unsafe driving behaviors in real-world data\". \"Overcomes\" is a strong word considering that it still does not seem fully reliable, especially in the behavior of the other traffic agents. This does not seem like a solved problem. Consider softening this.\n\nQUESTIONS:\nControllability: a) In the second-to-third image transition of Fig.1b, it seems like your model also went forward instead of turning more left? Was the action turn left during the whole sequence or just the first image? b) It is difficult to get an understanding of how controllable your WM is as the Trajectory Difference metric is I think an L2 loss over only a 2s prediction window. It would have been helpful to include some data on the drift in meters over longer predictions (the IDM itself has drift so it's imperfect but better than nothing).\n\nOther agents: This uses randomized driving signals mainly for the ego vehicle, but what about the other agents? in the attached video nuscenes1 (nice that this was included!) it seems like driving forward makes it so the car in front never tried to stop when it otherwise actually did. Am I correct in assuming that the \"distort reality to a nominal scenario\" problem of WM prediction that you tried to address for the ego motion is still present when it comes to how other cars behave? I think you might be hinting at this problem in your limitations section but can you be a bit more clear?\n\nCARLA artifacts?: Since the non-nominal situations are from CARLA, do you see the WM predictions looking more like CARLA / low-fidelity the longer you predict or the more you deviate?\n\nLIMITATIONS:\nMostly there, see comments above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}