Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
Authors: Shenyuan Gao
Jiazhi Yang
Li Chen
Kashyap Chitta
Yihang Qiu
Andreas Geiger
Jun Zhang
Hongyang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: World Model
Autonomous Driving
Video Prediction
Primary area: machine_vision

Abstract: World models can foresee the outcomes of different actions, which is of paramount importance for autonomous driving. Nevertheless, existing driving world models still have limitations in generalization to unseen environments, prediction fidelity of critical details, and action controllability for flexible application. In this paper, we present Vista, a generalizable driving world model with high fidelity and versatile controllability. Based on a systematic diagnosis of existing methods, we introduce several key ingredients to address these limitations. To accurately predict real-world dynamics at high resolution, we propose two novel losses to promote the learning of moving instances and structural information. We also devise an effective latent replacement approach to inject historical frames as priors for coherent long-horizon rollouts. For action controllability, we incorporate a versatile set of controls from high-level intentions (command, goal point) to low-level maneuvers (trajectory, angle, and speed) through an efficient learning strategy. After large-scale training, the capabilities of Vista can seamlessly generalize to different scenarios. Extensive experiments on multiple datasets show that Vista outperforms the most advanced general-purpose video generator in over 70% of comparisons and surpasses the best-performing driving world model by 55% in FID and 27% in FVD. Moreover, for the first time, we utilize the capacity of Vista itself to establish a generalizable reward for real-world action evaluation without accessing the ground truth actions.

Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
Shenyuan Gao 1.2 Andreas Geiger
Jiazhi Yang²
Li Chen
Kashyap Chitta Yihang Qiu? Hongyang Li
Jun Zhang":
Hong Kong University of Science and Technology
2 *OpenDrivel Lab at Shanghai AI Lab University of Hong Kong
University of Tübingen 'Tubingen AI Center
Code and model: github. com/ hwt///eDDrieeab//vssrr.e
Demo page: opendrivelab com/Vista
Abstract
World models can foresee the outcomes of different actions, which is of paramount importance for autonomous driving. Nevertheless, existing driving world models still have limitations in generalization to unseen enviromments, prediction fidelity of critical details, and action controllability for flexible application. In this paper, we present Vista, a generalizable driving world model with high fidelity and versatile controllability. Based on a systematic diagnosis of existing methods, we introduce several key ingredients address these limitations. To accurately predict worl-world dynamics at high resolution, we propose two novel losses to promote the learning of moving instances and structural information. We also devise effective latent replacement approach to inject historical frames as priors for coherent long-horing -orriio rollouts. For action controllability, we incorporate versatile set of controls from high-level intentions (command, goal point) to low-level maneuvers (trajectory, angle, and speed) through efficient learning strategy. After large-scale training, the capabilities of Vista can seamlessly generalize to different scenarios. Extensive experiments multiple datasets show that Vista outperforms the most advanced video generator in over 70% of comparisons and surpasses the best-performing driving world model by 55% FID and 27% in FVD. Moreover, for the first time, we utilize the capacity of Vista itself to establish a generalizable reward for real-world action evaluation without accessing the ground truth actions.
general-pure
Introduction
Driven by scalable learning techniques, autonomous driving has made encouraging strides over the past few years [18, 58. 135]. However, intricate and out-of-distribution situations are still intractable for state-of-the-art techniques [83]. One promising solution lies in world models [57, 76], which infer the possible future states of the world from historical observations and alternative actions, in turn assessing the feasibility of such actions. They hold the potential to reason with uncertainty and avoid catastrophic emrors [54, 76, 127], thereby promoting generalization and safety in autonomous driving.
Although a primary prospect of world models is to enable the generalization ability to novel environ- ments, existing driving world models are still constrained by data scale [90, 125, 127, 143, 147] and geographical coverage [54, 61]. As summarized in Table 1 and Fig. 1, they are also often confined to low frame rates and resolutions, resulting in a loss of critical details. Furthermore, most models only support single control modality such as the stecring angle and speed. This is insufficient to express various action formats ranging from high-level intentions to low-level maneuvers, and incompatible with the outputs of prevalent planning algorithms [12, 14, 21, 56, 58, 64]. In addition, generalizing action controllability unseen datasets understudied. These limitations impede the applicability of existing works, making it imperative to develop world model that overcomes these limitations.
Primary contact to Shenyuan at sygao@connect. ust. EEqual advising.
hk
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

Table 1: Real-world driving world models. Trained on large-scale high-quality driving data, Vista
performs at high spatiotemporal resolution and supports versatile action controllability. Private data
Model Setups
Action Control Modes
Method
Duta Scale
Frume Rale
Resolation
Sngle&Sppedd
Trajectory
Command
Goal Point
DriveSim [102]
7h
5 Hz
80x 160
DriveGAN
160h
8 lI
256x256
DriveDreamer [125]
5h
12 Hz
128x192
Drive WM [127]
5h
2 l
192x3384
VoCon [90]
5h
2 Hz.
256x4488
ADriver-I [61]
300h
2 Hz
256x512
GenAD [136]
2000h
256x448
GATA-1 [54]
4700h
25 11
x58x
Vista (Ours)
1740h
10
576x 1024
60x160
120x192
192x334
256o4455
250x
285-512
DrieDreemee
1201 64586
(2023/111
DiveGAN (2391/06
WovooSn 0022//12
GenAD (1924/00
12023/091
Vieta [oun)
Figure I: Resolution comparison. Vista predicts at a higher resolution than previous literature.
To this end, we introduce Vista, a driving world model that is proficient in cross-domain generalization, high-fidelity prediction, and multi- modal action controllability. Specifically, develop the predictive model on large corpus of worldwide driving videos [136] to foster its generalization ability. To enable coherent future extrapolation, we condition Vista three essential dynamic priors (Sec. 3.1). Instead of solely relying on the standard diffusion loss [5], we introduce two explicit loss functions to enhance dynamics and preserve structural details (Sec. 3.1), promoting Vista's 's ability to simulate realistic futures at high resolution. For flexible controllability, we incorporate a versatile set of action formats, including both high-level intentions such as commands and goal points, as well low-level maneuvers like trajectories, steering angles, and speeds. These action conditions are injected via unified interface, which is learned through an efficient training strategy (Sec. 3.2). Consequently, as Fig, 2 shows, Vista acquires the ability to anticipate realistic futures at 10 Hz and 576x 1024 pixels, and obtains versatile action controllability across various levels of granularity. We also demonstrate the potential of Vista as generalizable reward function to evaluate the reliability of different actions.
Our contributions are three-fold: (1) We present Vista, generalizable driving world model that can predict realistic futures at high spatiotemporal resolution. Its prediction fidelity is greatly improved by two novel losses that capture dynamics and preserve structures, along with exhaustive dynamic priors to sustain consistency in long-horizon rollouts. (2) Propelled by an efficient learning strategy, integrate versatile action controllability into Vista through unified conditioning interface. The action controllability of Vista can also generalize to different domains in zero-shot manner. (3) We conduct comprehensive experiments across multiple datasets verify the effectiveness of Vista. outperforms the most competitive general general-purpost purpose video generator and sets a new state-of-ther -fthe-o on nuScenes. Our empirical evidence shows that Vista can be used reward function to assess actions.
2 Preliminary
We initialize Vista with the pretrained Stable Video Diffusion (SVD) [5], latent diffusion model for image-to-vides generation. For sampling flexibility, SVD adopts centinuous-timestee formula [66, 111]. It converts data samples z to noise n through a diffusion process p(n|æz) ~ (xx o²I), and generates new samples by progressively denoising the latent towards do from Gaussian noise.
The training of SVD can be simplified to minimizing Ez,c.n Do - æ1| where Dg is
parameterized UNet denoiser and A, is re-weighting function omitted hereinafter for brevity. Based on this framework. SVD processes sequence of noisy latent {nz, ( 7l and generates a video with K = 25 frames. The generation process is guided by condition image, whose latent is concatenated channel-wise to the inputs, serving reference for content generation.
IR xxxxxx
Despite the high aesthetic quality, SVD lacks several key properties to function as a driving world model. As shown in Sec. 4, the first frame predicted by SVD is not identical to the condition image, making impractical for autoregressive rollout due to content inconsistency. addition, SVD struggles with the intricate dynamics of driving scenarios, entailing implausible motions. Moreover,

---

(A) High-Fidelity Future Prediction
Ds
15p
(B) Coherent Long-Horizon Rollout
Reward: 0.872 0.815
Reward: 0.870 0.849
"go forward'
Reward: 0.872 O
Reward: 0.855 860
(C) Versatile Action Controliability
(D) Generalizable Reward
Figure 2: Capabilities of Vista. Starting from arbitrary environments, Vista anticipate realistic and continuous futures at high spatiotemporal resolution (A-B). It can be controlled by multi-modal actions (C), and serve generalizable reward function evaluate real-world driving actions (D).
SVD cannot be controlled by any action format. In contrast, aim to build generalizable driving world model that predicts high-fidelity futures with realistic dynamics. It ought to be continuously extendable long horizons and flexibly controllable by multi-modal actions as illustrated in Fig. 2.
Learning a Generalizable Driving World Model
As depicted in Fig. 3, Vista adopts a two-phase training pipeline. First, we build a dedicated predictive model, which involves latent replacement approach to enable coherent future prediction and two novel losses to enhance fidelity (Sec. 3.1). To ensure the generalization to unseen scenarios, we utilize the largest public driving dataset [136] for training. In the second phase, incorporate multi-modal actions to learn action controllability with an efficient and collaborative training strategy (Sec. 3.2). Using the ability of Vista, further introduce generalizable approach to evaluate actions (Sec. 3.3).
3.1 Phase One: Learning High-Fidelity Future Prediction
Basic Setup. Since world models initiated to predict futures from the current state, the starting of their prediction should be firmly aligned with the condition image. Therefore, we tailor SVD into a dedicated predictive model by imposing the first frame the condition image and discarding the noise augmentation [5. 49] during training. With this prediction ability, Vista can perform long-term rollouts by iteratively predicting short-term clips and resetting the condition image with the last clip.
Dynamic Prior Injection. Nevertheless, using the aforementioned setup for training often results in irrational dynamics with respect historical frames, especially in long-term rollouts. We conjecture that this mainly arises from the ambiguity caused by insufficient priors about the tendency future motions, which is also common limitation of existing driving world models [54, 68, 125, 127, 136].
Estimating coherent futures requires at least three essential priors that inherently govemn the future motion of instances in the scene: position, velocity, and acceleration. Since velocity and acceleration are the first- and second-order derivative of position respectively, these priors can be entirely derived by using three consecutive frames for conditioning. Concretely, we build frame-wiss mask m E (0. 1) with length of K to indicate the presence of condition frames. The mask is set sequentially following the time order, with most three elements being assigned to denote three condition

---

prediction
time
Video rrpll
Vista
video oulput
outoregressive fou!
Vista
training phose
dynomii priors
zero-ini pro projection
video
Vista
LoRA
video output
loscest
Inpul
projection
initial framm
STOP
oction impul
EoRmond poc
mulli modaa actan control
training phose
Figure 3: [Left]] Vista pipeline. In addition to the initial frame. Vista can absorb more priors about future dynamies via latent replacement. Its prediction can be controlled by different actions and be extended to long horizons through autoregressive rollouts. [Right]: Training procedure. Vista takes two training phases, where the second phase freezing the pretrained weights to learn action controls.
frames. Instead of concatenating additional channels to the inputs, we inject new condition frames by replacing the corresponding noisy latent nt, with the clean latent 2; encoded by the image encoder. Formally, the input latent constructed the clean latent, we duplicate new timestep embedding from the pretrained weights and allocate it to the condition frames according to m. The timestep embeddings for condition frames and prediction frames are trained separately. Compared to channel -wise concatenation, we find that replacing the latent more effective and flexible in absorbing varying numbers of condition frames. In addition, we observe that the replaced latent, when applied to SVD directly, does not degrade its generation quality. Thus, the original performance will not be disturbed when the training launched. Since there is no need to predict the observed condition frames, we exclude them from the loss as follows:
m z =+
- (see Fig. 3 [Left]). To discern
K E m.) @ (a((fo;)) - zil² -xil))
Lditfusion EEx,,
where Dg is the UNet denoiser that shares the same architecture with SVD. With the replaced latent holding sufficient priors, Vista can fully capture the status of the surrounding instances and predict more coherent and plausible long-term futures through iterative rollouts. In practice, leverage the last three frames of a predicted clip as dynamic priors for the next prediction step during rollouts.
Dynamics Enhancement Loss. Unlike general videos that cover rather small spaces, driving videos capture much larger scenes [136]. In most driving videos, distant and monotonous regions dominate the view, with the moving foreground instances only occupying a relatively small area [17]. However, the latter often exhibit higher stochasticity, complicating their prediction. Since Eq. (1) supervises all outputs uniformly, it cannot effectively discriminate the nuances of different regions as Fig. 4(b) shows. As result, the model cannot efficiently learn to predict realistic dynamics in crucial regions.
As the discrepancy between two adjacent frames provides considerable motion patterns [123, 132], we introduce an additional supervision to encourage the learning of dynamics for crucial regions. To be specific, we first introduce dynamics- aware weight {we, Wy, ..., w&} E R - XxW wy,..., that highlights the regions where the prediction has inconsistent motion compared to the ground truth:
W = [((aa(f;;)) σ) (fff-1ooo) - (2. 2=11
(2)
For numerical stability, we normalize w within each video clip. As shown in Fig. 4(c), the weight amplifies the presence of large motion disparities, highlighting dynamic regions while excluding monotonous backgrounds. Given the causality of future prediction, iee subsequent frames ought to follow previous ones, define new loss by penalizing the latter frame of each adjacent frame pair:
Caynamics
Σ sg(u;) (1 11-mm m4) o (D -2,11
where sg(-) stops the gradient. By adaptively re- weightig the standard diffusion loss, Ldymamies can boost the learning efficiency of dynamic regions, e.g., the moving vehicles and sidewalks in Fig. 4(d).
Structure Preservation Loss. The trade-off between perceptual quality and motion intensity has been widely acknowledged in video generation [3, 32, 73, 144], and our case is no exception. When

---

[al
[b] |c]
original slandard dittusion dynamics
decoded fectures
Figure 4: Illustration on loss design. Different from the standard diffusion loss (b) that is distributed uniformly, dynamics enhancement loss (d) enables adaptive concentration critical regions (c) (e.g., moving vehicles and roadsides) for dynamics modeling. Moreover, by explicitly supervising high-frequency features (e), the learning of structural details (e.g., edges and lanes) can be enhanced.
it comes to nigh-resolution prediction for dynamic driving scenarios, we discover that the predicted structural details degrade severely with wer-smoothed or broken objects, e.g., the outlines of vehicles unravel quickly as they move (see Fig. 12). To alleviate this problem, is important to place more emphasis on structural details. Based the fact that structural details, such edges and textures, mainly reside in high-frequency components, we identify them in the frequency domain as follows:
F(2)) IFFT (H O FFT();)).
where FFT and IFFT are the 2D discrete Fourier transform and inverse discrete Fourier transform respectively, and H is ideal 2D high-pass filter that truncates low-frequency components under a certain threshold. The Fourier transforms are applied on each channel of z; independently. As illustrated in Fig. 4(e), features associated with structural information can be effectively emphasized by Eq. (4). The corresponding features from the predicted latent Do (ffA; can also be extracted similarly. With the extracted high-freyuency features, we devise new structure preservation loss as:
Strructure Eg,o,
(1-m.)) (F)()(()5))) -
This loss function minimizes the disparity of high-frequency features between prediction and ground truth, so that more structural information can be retained. Our final training objective a weighted sum of Eq. (1), Eq. (3) and Eq. (5), where a and X2 are trade-off weights to balance the optimization:
Linal = Laiffusion + Cdynamies + A2 Ctt
3.2 Phase Two: Learning Versatile Action Controllability
Unified Conditioning of Versatile Actions. To maximize usage flexibility, a driving world model should be able leverage multiple action formats with different characteristics. For instance, one may use the world model to evalute high-level policies [127], or to execute low-level maneuvers [102]. However, existing approaches only support limited action controls [54, 61, 90, 125, 127], inhibiting their flexibility and applicability. Therefore, we incorporate versatile set of action modes for Vista: (1) Angle and Speed stand for the utmost fine-grained action controls. We normalize angles to 1-1, and represent speeds in km/ h. The signals from different timestamps are concatenated sequentially. (2) Trajectory is a series of 2D displacements in ego coordinates. It is widely used as the output of planning algorithms [12. 21, 58, 62, 63]. We represent the trajectory in meters and flatten it into a sequence. (3) Command is the most high-level intention. Without loss of generality, we define four commands, i.e. go forward, turn right, turn left, and stop, which are implemented as categorical indices. (4) Goal Point is a 2D coordinate projected from the short-term ego destination onto the initial frame, serving as interactive interface [74]. The coordinate is normalized by the image size.
Note that these actions are heterogeneous and cannot be used interchangeably. After transforming all these actions into numerical sequences, encode them unified concatenation of Fourier embeddings [114, 116) (see Fig. 3). These embeddings can be jointly ingested by learning additional projections to expand the input dimension of the cross-attention layers in the UNet [5]. The new
5

---

projections are initialized as zeros to enable gradual learning from the pretrained state. We empirically discover that incorporating action conditions through cross-attention layers yields faster convergence and stronger controllability compared to other approaches such as additive embeddings [128, 136).
Efficient Learning. We learn action controllability after the first training phase. Since the number of total iterations is crucial for diffusion training [5, 22, 32, 99], we separate action control learning into two stages. In the first stage, we train our model at a low resolution (320x576), which achieves 3.5 higher training throughput compared to the original resolution (576x 1024). This stage constitutes the majority of training iterations. Then, we finetune the model at the desired resolution (576x 1024) for a short duration, so that the learned controllability can effectively cater to high- nigh-resolution prediction.
However, tuning the UNet [5] at a lower resolution directly may undermine the high-fidelitg prediction ability. Conversely, freezing all UNet weights and training the new projections alone would precipitate quality decline (see Appendix D), suggesting the necessity to make the UNet adaptable. To solve this, we freeze the pretrained UNet and introduce parameter-efficient LoRA adapters [55] to each attention layer. After training, the low-rank matrices can be seamlessly integrated with the frozen weights, without introducing extra inference latency. Thus, the pretrained weights remain intact when training at the low resolution, avoiding deterioration of the pretrained high-fidelity prediction ability.
Since the parameters of the camera and vehicle are unavailable for weenwworld scenarios, it seems impossible to obtain multiple equivalent action conditions simultancously at inference time. Addition- ally, it will entail prohibitively expensive training to encompass all possible combinations of action conditions. Hence, unlike common practices that activate all conditions during training, we enforce the independence of different action formats by enabling only one of them for each training sample. The remaining action conditions will be filled with zeros as unconditional inputs. As demonstrated in Appendix D. this simple constraint prevents the squandering of training cost on action combinations and maximizes the learning efficiency of each individual action mode within the same training steps.
Collaborative Training. Note that the aforementioned action conditions are not available in OpenDV- YouTube [136]. On the other hand, nuScenes [10] has adequate annotations to derive these conditions. To maintain generalization and learn controllability in tandem, we introduce a collaborative training strategy by utilizing the samples from both datasets, with the action conditions for YounDV-YuuTun set to zero The action control learning phase adopts the same loss as Eq. (6). By learning from two complementary datasets, Vista gains versatile controllability that are generalizable to novel datasets.
3.3 Generalizable Reward Function
One application of world models is to evaluate actions by engaging a reward module [40, 42, 43, 76]. Drive-Wie [127] establishes reward using external detectors [82, 84]. However, these detectors are developed on a particular dataset [10], which may become a bottleneck for reward estimation in arbitrary scenarios. On the other hand, Vista has ingested millions of human driving logs, exhibiting strong generalization across scenes. Based on the observation that out-of-distribution conditions will lead to increased diversity in generation [28, 60], we utilize the prediction uncertainty from Vista itself as the source of our reward. Different from MM,, our reward function seamlessly inherits the generalization of Vista without resorting to external models. Specifically, we estimate uncertainty via conditional variance. For reliable approximation, we denoise from randomly sampled noise with the same condition frame and action a for M rounds. Our reward function R(c, a) is then defined as the exponential of averaged negative conditional variance:
n ED (m) (n:c,a)) (m;ccn
R(c, a) eep avg
M (m)
1
(n; - -()))]
(8)
where avg(.) averages all latent values within the video clip. Based on this formulation, unfavorable actions with larger uncertainties will lead to lower rewards. In contrast to commonly used evaluation protocols (e.g., the 1.2 error), our reward function can evaluate actions without referring to the ground truth actions. Note that we do not normalize the estimated rewards for the simplicity of definition, but it is straightforward to amplify the relative contrast by rescaling the estimated rewards with factor.

---

Table 2: Comparison of prediction fidelity on nuScenes validation set. Vista achieves encouraging
results that outperform the state-of-the-art driving world models with a significant performance gain.
DriveGAN [02
DriveDreamer
WoVoGen
Drive-WM [127]
GenAD
Metric
[125]
[136]
(Ours)
FID 1
73.4
52.6
27.6
15.8
15.4 184.0
6,9
FVD.
502.3
452.0
417.7
122.7
89,4
comppttd
conditin
Figure 5: Driving futures predicted by different models using the same condition frame. We contrast Vista publicly available video generation models using their default configurations. Whilst previous models produce misaligned and corrupted results, Vista does not suffer from these caveats.
Altivved (20 256cc5 3.50)
Drive WDM 12Hz 192-844 Bil
Vista (10Hz. 576x 1024. 15s)
SVD
Figure 6: [Top]: Long-horizon prediction. Vista can forecast 15 seconds nigh-resolution futures without much degradation, encompassing long driving distances. The length of the blue lines indicate the duration the longest prediction showeased by previous works. [Boftom]: Long-term extension results of SVD. SVD fails to generate consistent high-fidelity videos autoregressively as Vista does.
Experiments
In this section, we first demonstrate Vista's strengths generalization and fidelity in Sec. 4.1. We then show the impact of action controls in See. 4.2. We also substantiate the efficacy of the proposed reward function in Sec. 4.3. Finally, we conduct ablation studies on our key designs in Sec. 4.4. For more implementation details and experimental results, please refer to Appendix C and Appendix D.
4.1 Comparisons of Generalization and Fidelity
Automatic Evaluation. Since none of the driving world models publicly accessible, compare these methods with their quantitative results on nuScenes. We filter 5369 valid samples from the validation set to conduct FID [47] and FVD [115] evaluation. For FID evaluation, the predicted frames to the resolution of 258x For FVD evaluation, we use all 25 frames each video clip and downsample them to 224 224 following LVDM [46]. Table 2 reports the results of all methods. In both metrics, Vista surpasses previous driving world models with a considerable margin.
crop and resize
Human Evaluation. against three prominent conpoal-purpose video generators trained on web-scale data [5, 133, 144] (see Fig. 5). It is known that automatic metrics like FVD [115] cannot conclusively reveal perceptual
To analyze the generalization of Vista across different datasets, we compare it

---

Visual Quality
Motion Rationality
Average
action free
367
96.67
23.33
222.7
$4.25
w/ command
292:2
72.75
on speed
2300
trojectory
3.33
6.67
21.22
15:15
Averoge Subset PVD 4 on Waymo
ZVGen.X
DynomiCratter
DynamCronter
w/ command
273.7
Baselines
Irojectory
Figure 7: Human evaluation results. The value denotes the percentage of the times that one model is preferred over Applying action controls will produce the other. Vista outperforms existing works in both metrics. more similar predictions the real data.
Figure 8: Efficacy of action controls.
tralectory
commond
'st00'
fumn loff
Figure 9: Versatile action controllability. Vista can predict the corresponding outcomes in response multi-modal action conditions across diverse scenarios. More results can be found in Appendix
quality [3, 6, 32, 130, 136], let alone real-world dynamics. Therefore, we opt for human evaluation for more faithful analysis. Following recent advances [3, 5, 6, 15, 16, 32, 122, 126], adopt the Two-Alternative Forced Choice protocol. Specifically, participants are presented with a side-by-side video pair and asked to choose the video they deem better on two orthogonal aspects: visual quality and motion rationality. To avoid potential bias, we crop each video to fixed aspect ratio, downsample them the same resolution, and trim the excess frames when Vista generates longer videos than others. We only feed one condition frame to align with other models. To ensure the variety of scenes, we uniformly assemble 60 scenes from four representative datasets: OpenDV- Youuuooouubb [136]. nuScenes [10], Waymo [112]. and CODA [79]. These datasets collectively exemplify the intricacy includes geofenced districts, Waymo offers unique domain compared to our training data, and CODA contains extremely challenging corner cases. We collect total of 2640 answers from 33 participants. As presented in Fig. 7, Vista outperforms all baselines on both aspects, demonstrating its profound comprehension of the driving dynamics. Further, unlike other models that are only applicable for short-term generation, Vista can accommodate more dynamic priors and produce coherent long-horizon rollouts as shown in Fig. 6.
and diversity of real-ve world driving, e.g., OpenDV-
4.2 Results of Action Controllability
Quantitative Results. To evaluate the impact of action controls, we divide the validation set of both nuScenes and the unseen Waymo dataset into four subsets according then generate predictions using different modalities of the ground truth actions. The FVD score is measured on each subset and then averaged. A lower FVD score reflects closer distribution to the ground truth videos, indicating that the predictions exhibit more resemblance to each specific type of behavior. Fig. 8 shows that action controls emulate the corresponding movements effectively.
command categories. We
We also introduce new metric named Trajectory Difference to assess control consistency. Following GenAD [136], we train an inverse dynamics model (IDM) that estimates the corresponding trajectory from a video clip. An illustration of IDM is shown in Fig. 13. We then send Vista's prediction the IDM and calculate the L2 difference between the ground truth trajectory and the estimated trajectory.

---

ground
Bron Rowerd:
fruth
Actiond La Bron 36 Rcword:0.90
A40000 Keood
with prior
with
priors
with
priors
12 Enior
Figure 10: [Left]: Average reward on Waymo with different L2 errors. [Right]: Case study. The relative contrast of our reward can properly assess the actions that the L2 enror fails judge.
Figure 11: Effect of dynamic priors. Injecting more dynamic priors yields more consistent future motions with the ground truth, such as the motions of the white vehicle and the billboard on the left.
Effect of Dynamies Enhancement Loss
Effect of Structure Preservation Loss
SO CampTaae
wio dynamics
statfrrme
w/ dymamics lets
Figure 12: [Left]] Effect of dynamics enhancement loss. The model supervised by the dynamics enhancement loss generates more realistic dynamics. In the first example, instead of remaining static, the front car moves forward normally. In the second example, when the ego-vehicle steers right, the shift towards the left naturally adhering to the real-world geometric rules. [Right]: Effect of structure preservation loss. The proposed loss yields a clearer outline of the objects as they move.
The differences are measured over 2 seconds. The lower the trajectory difference, the stronger the control consistency Vista exhibits. We conduct the experiments on nuScenes and Waymo. For each dataset, collect subset that contains 537 samples. As reported in Table 3, Vista can be effectively controlled by different modalities of actions, resulting in more consistent motions to the ground truth.
Qualitative Results. Fig. 9 exhibits the versatile action controllability of our model. Vista can be effectively controlled by modal actions, even in unseen scenarios beyond the training domain. In Appendix E, we also showcase the counterfactual reasoning ability of Vista using abnormal actions.
4.3 Results of Reward Modeling
To validate the efficacy of our reward function, we jitter the ground truth trajectories into series of inferior trajectories. Specifically, we compute the standard deviation of each waypoint from the nuScenes training set prior distributions. These priors are jointly rescaled sample perturbations with different 1.2 errors. The perturbations are then added as offsets to the ground truth trajectories. To ensure the plausibility sampled trajectories, we adopt an explicit correlating strategy [35, 95] to regularize off'set sampling and recursively sample new trajectories until their offsets are consistent in tendencies. To demonstrate the generality of our reward function, conduct reward estimation on Waymo [1]2], which is unseen in training. This is done by uniformly sampling from each command category on Waymo validation set, resulting in 1500 cases total. We compare the average reward of the trajectories with varying L2 errors in Fig. 10. Our reward decreases when the deviation from ground truth increases, underscoring the potential of our approach to serve as a viable reward function. It also holds the promise to remedy the irrationality in current evaluation protocols for planning [18, 83, 141], such as the 1.2 error shown in Fig. 10. More in-depth analysis of rewards, including sensitivity to hyperparameters and reward of other actions, provided in Appendix D.

---

Table 3: Impacts of different action conditions and dynamic priors. By applying action conditions and dynamic priors, Vista can predict motion that is more consistent compared the ground truth.
Average Trajectory Difference
Dataset
Condition
with prior
with priors
with 3 priors
GT video
0.379
0.379
0.379
action-fre free
3,785
2.597
1.820
goal point command
2,869
2.192
1.585
nuScenes
3,129
2.403
1.593
+ angle & speed
1,562
1.123
0.832
+ trajectory
1,559
1.148
0.835
GT video
0.893
0.893
0.893
action-free
3,646
2.901
2.052
Waymo
+ command
3,160
2.561
1,902
trajectory
1.187
1.147
1.140
4.4 Ablation Study
Fig. 11 visualizes the outcomes of using different orders of dynamic priors. The order of priors corresponds to the number of condition frames. It shows that dynamic priors play pivotal role in long-horizon rollouts, where the coherence with respect to historical frames is essential.
Dynamic Priors.
To further demonstrate the efficacy of dynamic priors, conduct a quantitative evaluation in Table 3. Specifically, we use the IDM in Sec. 4.2 infer the trajectories the predicted videos with different orders of dynamic priors. The diminishing differences in trajectory suggest that introducing more priors can effectively improve the consistency between prediction and ground truth.
future prediction
Dynomics Mcdel
Vista
Pred
Auxiliary Supervisions. To verify the effectiveness of the two losses proposed in Sec. 3.1, we devise two additional variants by individually ablating each loss from variant that incorpo- rates both losses. We qualitatively compare their effects through Fig. 12, which confirms that the dynamics enhancement loss promote the learning of real-world dynamics, and the structure preservation loss can reinforce the prediction of structural details.
Inawerkoyy Diforence
sejectory, commandd. gool point.
mullimocial octon contros
GT Trch
hatorry
Figure 13: An illustration of the IDM experiments Table 3.
5 Conclusion
In this paper, introduce Vista, generalizable driving world model with enhanced fidelity and controllability. Based on our systematic investigations, Vista is able to predict realistic and continuous futures high spatiotemporal resolution. It also possesses versatile action controllability that is generalizable to unseen scenarios. Moreover, it can be formulated as a reward function to evaluate actions. We hope Vista will usher in broader interest in developing generalizable autonomy systems.
Limitations and future work. As an early endeavor, Vista still exhibits some limitations with respect to computation efficiency, quality maintenance, and training scale. Our future work will look into applying our method to scalable architectures [54, 97]. More discussions are included Appendix A.
Acknowledgments
This work is supported by National Key R&D Program of China (2022Z00160104 National Natural Science Foundation of China (622061 72), and Shanghai Committee of Science and Technology (23YF1462000). This work is also partially supported by the BMBF (Tübingen AI Center, FKZ: 011S18039A), the DFG (SFB 1233, TP 17, project number: 276693517), and the EXC (number 2064/1 - project number: 390727645). We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Kashyap Chitta. We also appreciate Zetong Yang, Chonghao Sima, Linyan Huang, and the rest members from OpenDrivellab for valuable feedback.
We express sincere gratitude to all anonymous participants for helping with the human evaluation.
10

---

References
[1] Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenhaum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional Foundation Models for Hierar- chical Planning. In NeurIPS, 2023. 19
[2]
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video Pretraining (VPT): Learning to Act by Watching Unlabeled Online
Videos. In NeurlPS, 2022. 19
[3] Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang. Deqing Sun. Tali Dekel, and Inbar Mosseri. Lumiere: A Space-Time Time Diffusion Model for Video Generation. arXiv preprint arXiv:2401.129% 2024. 4,
[4] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models. In ICLR, 2024. 19
[5] Andreas Blattmann, Tim Dackhorn Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling Latent Video Diffusion Models Large Datasets, arXiv preprint A.Ki:2331.11111
2023. 2, 3, 6, 8, 21, 22, 23, 24, 26
[6] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim. Sanja Fidler, and Karsten Kreis. Align Your Latents: Hfigh-Resolution Video Synthesis with Latent Diffusion Models. In CVPR, 2023. 20. 21
[7]
Daniel Bogdoll, Yitian Yang, and J Marius Zöllner. MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations. arXiv preprint arX1v:2311.1176. 2023. 21
[8] Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to Follow Image
Editing Instructions. In CVPR, 2023. 20
[9] Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richic Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanic Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zoina, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel. Genie: Generative Interactive Environments. arXiv preprint arXiv:240..1539), 2024. 19, 20, 21
[10] Holger Caesar, Varun Bankiti, Alex H. I Lang, Sourahh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yuxin Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A Multimodal Dataset for Autonomous Driving. In CVPR, 2020. 6, 8, 20, 21, 23, 26
[11] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuPlan: A Closed-Loop ML-based Planning Benchmark for Autonomous Vehicles. In CVPR Workshops, 2021. 20
[12] Sergio Casas, Abbas Sadat, and Raquel Urtasun. MP3: A Unified Model to Map. Perceive, Predict and Plan. In CVPR, 2021. 5, 21
[13] Jun Cen, Chenfei Wu, Xiao Liu, Shengming Yin, Yixuan Pei, Jinglong Yang. Qifeng Chen, Nan Duan, and Jianguo Zhang. Using Left and Right Brains Together: Towards Vision and Language Planning. arXiv preprint ar arXiv:2402. 10534, 2024. 19
[14] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühhl. Learning by Cheating. CoRL, 2019.
1,
[15] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafterI: Open Diffusion Models for High-Quan Sugh-Qualtty Video Generation. arXiv preprint arXiv:2310. 19512, 2023. 8, 21
[16] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models. In CVPR, 2024.
[17] Kai Chen, Faze Xie, Zhe Chen, Yibo Wang, Lanqing Hong. Zhenguo Li, and Dit-Yan Yeung. GeoD- iffuuioo: Text-Prompted Geometric Control for Object Detection Data Generation. In ICLR, 2024.
[18] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-End Autonomous Driving: Challenges and Frontiers. arXiv preprint arXiv:2306. /6927, 2023. 1,9, 19
[19] Shoufa Chen, Mengmeng Xu, Jiawei Ren. Yuren Cong. Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan- Manuel Perez- Rua. GenTron: Delving Deep into Diffusion Transformers for Image and Video Generation. In CVPR, 2024. 20
11

---

[20] Kashyap Chitta, Daniel Dauner, and Andreas Geiger. SLEDGE: Synthesizing Simulation Environments for Driving Agents with Generative Models. arXiv preprint arXiv:2403.1753. 2024. 21
[21] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Trans- Fuser: Imitation with Transformer-based Sensor Fusion for Autonomous Driving. JEEE TPAMI, 2023. 1, 21
[22] Xiaoliang Dai. Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, and Devi Parikh. Emu: Enhancing Image Generation Models using Photogenic Needles in Haystack. arXiv preprint arXiv:2309. /5807, 2023.
[23] Daniel Dauner. Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang. Hongyang
Li. Igor Gilitschenski. Boris Ivanovic. Marco Pavone, et al. NAVSIM: Data- Driven Rea-Reeet Autonomous Vehicle Simulation and Benchmarking. arXiv preprint arXiv:2406. /5349, 2024. 20
[24] Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis. In NeurlPS,
2021. 21
[25]
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning Universal Policies via Text-Guided Video Generation. In NeurlPS, 2023. 19
[26] Yilun Du, Sherry Yang, Pete Florence. Fei Xia. Ayzaan Wahid, brian ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tompson. Video
Language Planning. In ICLR, 2024. 19, 21
[27] Frederik Ehert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual Fore- sight: Model-based Deep Reinforcement Learning for Vision-based Robotic Control. arXiv preprint
1XX::18812.0068. 2018. 19
[28] Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Vidco Prediction Models as Rewards for Reinforcement Learning. In NeurlPS, 2023, 6, 19
[29] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for Righ-Resolutton Image Synthesis. In CVPR, 2021. 21
[30] Chelsea Finn and Sergey Levine. Deep Visual Foresight for Planning Robot Motion. In ICRA, 2017. 19,
21
[31] Zeyu Gao, Yao Mu, Ruoyan Shen, Chen Chen, Yangang Ren, Jianyu Chen, Shengbo Eben Li, Ping Luo, and Yanfeng Lu. Enhance Sample Efficiency and Robustness of End- to lo-End Urban Autonomous Driving via Semantic Masked World Model. In NewrIPS Workshops, 2022. 21
[32] Rohit Girdhar, Mannat Singh, Andrew Brown. Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu Video: Factorizing Text-to- Video Generation by Explicit Image Conditioning. arXiv preprint arXiv:2311. I. 10700, 2023, 4, 6, 8, 20, 21
[33] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In NewrIPS, 2014. 21
[34] Yuwei Guo. Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. AnimateDiff: Animate Your Personalized Text-to-lmage Diffusion Models without Specific Tuning, In ICLR, 2024. 21
[35] Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martin-Martin, and Li Fei-Fei. MaskViT: Masked Visual Pre- PrainTrinn for Video Prediction. In ICLR, 2023. 9, 19, 21, 23
[36] Tarun Gupta, Wenbo Gong. Chao Ma. Nick Pawlowski, Agrin Hilmkil, Meyer Scetbon, Ade Famoti, Ashley Juan Llorens. Jianfeng Gao, Stefan Bauer, Bernhard Kragic, Danica an Schölkopf, and Cheng Zhang. The Essential Role of Causality in Foundation World Models for Embodied Al. arXiv preprint vxXiv:2402.06660 2024, 25
[37] Wes Gurnee and Max Tegmark. Language Models Represent Space and Time. In ICLR, 2024. 21
[38] Nicholas Guttenberg and CrossLabs. Diffuusio with Offset Noise, 2023. 22
[39]
David Ha and Jürgen Schmidhuber. Recurrent World Models Facilitate Policy Evolution. In NeurlPS,
2018. 21
[40] Danijar Hafner, Timothy Lillicrap, Jimmy Ba. and Mohammad Norouzi. Dream to Control: Learning Behaviors by Latent Imagination. arXiv preprint 1:Xiv:1112.01003 2019. 6, 21
[41] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha. Honglak Lee, and James Davidson. Learning Latent Dynamics for Planning from Pixels. ICML, 2019. 21
[42] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with Discrete World Models. In ICLR, 2021. 6.21
12

---

[43] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering Diverse Domains through
World Models, arXiv preprint Stt1::2301.040044 2023. 6,
[44] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong. Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with Language Model is Planning with World Model. In EMNLP, 2023. 21
[45] Haoran He, Chenjia Bai, Ling Pan. Weinan Zhang. Bin Zhao, and Xuelong Li. Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning. arXiv preprint arXiv: 14407, 2024. 21
[46] Yingqing He, Tianyu Yang. Yong Zhang. Ying Shan, and Qifeng Chen. Latent Video Diffusion Models for High-Fidelity Long Video Generation. arXiv preprint arXiv:22) 1.32221, 2022. 7,21
[47] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NeurlPS, 2017. 7
[48] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In NeurlPS, 2020.
[49] Jonathan Ho, Chitwan Saharia, William Chan, David J Flect, Mohammad Norouzi, and Tim Salimans.
Cascaded Diffusion Models for High Fidelity Image Generation. JMLR, 2022. 3, 20, 22
[50] Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. arXiv preprint ar arXiv:2207.12590
2022. 22
[51] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.
Video Diffusion Models. arXiv preprint arXiv:2204.033558 2022, 21
[52] Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yco, Alex Kendall, Roberto Cipolla, and Jamic Shotton. Model-based Imitation Learning for Urban Driving. In NeurIPS, 2022. 21
[53] Anthony Hu, Zak Murez, Nikhil Mohan, Soffa Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras. In ICCV, 2021. 19,
[54] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. GAIA-I: A Generative World Model for Autonomous Driving, arXiv preprint ar3iv:2309. 17080, 2023. 1, 2, 3,5, 10, 20, 21
[55] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022. 6, 22
[56] Shengehao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning. In ECCV, 2022. 1,21, 22
[57] Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha. Seungchan Kim, Yaqi Xie, Tianyi Zhang, Shibo Zhao, Yu Quan Chong, Chen Wang, Katia Sycara, Matthew Johnson-Roberson, Dhruv Batra, Xiaolong Wang, Sehastian Scherer, Zsolt Kira, Fei Xia, and Yonatan Bisk. Toward General- Purpose Robots via Foundation Models: A Survey and Aeal--na arXiv preprint 2r1iv:2312.08788 2023. 1, 20
[58] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning- Oriented Autonomous Driving. In CVPR, 2023. 1, 2,1, 22
[59] Zhiting Hu and Tianmin Shu. Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning. arXiv preprint ar3iv:23 2023. 21
[60] Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffuusion Reward: Learning Rewards via Conditional Video Diffusion. arXiv preprint arXiv:2312.1413:4 2023. 6. 19
[61] Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. ADriver-I: A General World Model for Autonomous Driving. arXiv preprint arXh::2311.13549, 2023.
5,
[62] Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick I Langechuan Liu, and Hongyang Li. DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving. In ICCV, 2023. 5
[63] Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui Hc, Junchi Yan, and Hongyang Li. Think Twice Before Driving: Towards Scalable Decoders for End-to End Autonomous Driving. In CVPR, 2023.
5
[64] Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang. and Xinggang Wang. VAD: Vectorized Scene Representation for Efficient Autonomous Driving. In ICCV, 2023. 22.
13

---

[65] Michael I Jordan and David E Rumelhart. Forward Models: Supervised Learning with a Distal Teacher. Cognitive Science, 1992. 19
[66] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion-based Generative Models. In NeurlPS, 2022. 2, 22
[67] Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point Cloud Forecasting as a Proxy for
4D Occupancy Forecasting. In CVPR, 2023. 21
[68] Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. DriveGAN: Towards a Controllable High-Quality Neural Simulation. In CVPR, 2021. 2, 3, 21
[69] Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning Simulate Dynamic Environments with GameGAN. CVPR, 2020. 21
[70] Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes, arXiv preprint arXiv:1312.6114. 2013. 21
[71] Po-Chen Ko, Jiayuan Mao, Yilun Du Shao- Hua Sun, and Joshua B Tenenbaum. Learning to Act from Actionless Vidcos through Dense Corresoondences. In ICLR, 2024, 19
[72] Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A World
Model for Indoor Navigation. In ICCV, 2021. 21
[73] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Horuung, Hartwig Adam, Hassan Akhari, Yair Alon, Vighnesh Birodkar, Yong Cheng, Ming-Chang Chiu, Josh Dillon, Irfan Essa, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, David Ross, Grant Schindler, Mikhail Sirotenko, Kihyuk Sohn, Krishna Somandepalli, Huisheng Wang. Jimmy Yan. Hsuan Yang. Xuan Yang. Bryan Seybold, and Lu Jiang. VideoPoet: Large Language Model for Zero-Shot Video Generation. arXiv preprint arXiv:2312.1412. 2023. 4
Ming-Hsuan
[74] Hanyang Kong. Dongze Lian, Michael Bi Mi, and Xinchao Wang- DreamDrone. arXiv preprint arXiv:2312.085
2023. 5, 21
[75] Lei Lai, Zhongkai Shangguan, Jimuyang Zhang, and Eshed Ohn Ohn-Bar. XVO: Generalized Visual Odometry via Cross-Modal Self-Training. In ICCV, 2023. 19
[76] Yann LeCun. A Path towards Autonomous Machine Intelligence. Open Review, 62, 2022. 1, 6, 19, 20, 21
[77]
Hongyang Li, Yang Li, Huijie Wang, Jia Zeng, Huilin Xu, Pinlong Cai, Li Chen, Junchi Yan, Feng Xu, Lu Xiong, Jingdong Wang, Futang Zhu, Chunjing Xu, Tiancai Wang, Fei Xia, Beipeng Mu, Zhihui Peng. Dahua Lin, and Yu Qiao. Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future. arXiv preprint arXiv:2312.03444
2023. 21
[78) Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang. Jia Zeng. Zhiqi Li, Jiazhi Yang, Hamming Deng, Hao Tian, Enze Xie, Jiangwei Xie, Li Chen, Tianyu Li, Yang Li, Yulu Gao, Xiaosong Jia, Si Liu, Jianping Shi, Dahua Lin, and Yu Qiao. Delving Into the Devils of Bird's- Eyeeewww Perception: Review, Evaluation and Recipe. JEEE TPAMI, 2023. 21
[79] Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit-Yan Yeung, Xiaodan Liang. Zhenguo Li, and Hang Xu. CODA: A Real-World Road Corner Case Dataset for Object Detection Autonomous Driving. In ECCV, 2022. 8,26
[80] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. ICLR, 2023. 21
[81] Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Trink2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2). arXiv preprint arXiv:2402. 16720 2024. 21
[82] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. BEVFormer: Learning Bird's --ye View Representation from Multi-Camert Images via Spatiotemporal
Transformers. In ECCV, 2022. 6, 19
[83] Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong La, and Jose M Alvarez. Is Ego Status All You Need for Open-Loop End-to-End End Autonomous Driving? In CVPR, 2024. 9, 19, 21
[84] Beneheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction. In ICLR, 2023. 6. 19
[85] Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to Model the World with Language. arXiv preprint 0rX1v:2308.003399 2023. 21
[86] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World Model on Million-Lengle Video and Language With RingAtenntoon. arXiv preprint arXi::2402.08266 2024. 21
14

---

[87] Πly Loshchilov and Frank Hutter. Decoupled arXiv:1711.05001, 2017. 22
Weight Decay Regularization.
arXiv preprint
[88] William Lotter, Gabriel Kreiman, and David Cox. Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning In ICLR, 2017. 21
[89] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. In NewrIPS, 2022. 20
[90] Jiachen Lu, Ze Huang. Jiahui Zhang, Zeyu Yang, and Li Zhang. WoVoGen: World Volume-Aware Diffusion for Controllable Multi-Cameri Driving Scene Generation. arXiv preprint arXiv:2312.02954
2023. 2, 2.
[91] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent Consistency Models: Synthesizing Nigh-Resolution Images with Few-Step Inference. arXiv preprint arXiv:2310.04378 2023. 20
[92] Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured World Models from Human Videos. In RSS, 2023. 21
[93] Chenlin Meng. Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On Distillation of Guided Diffusion Models. In CVPR, 2023. 20
[94] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are Cample-Efficiee World Models, In ICLR, 2023. 21
[95] Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep Dynamics Models for Learning Dexterous Manipulation. In CoRL, 2020. 9,
[96] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-Conditional Video Prediction using Deep Networks in Atari Games. In NeurlPS, 2015. 21
[97] William Peebles and Saining Xie. Scalable Diffusion Models with Transformers, ICCV, 2023. 10. 20
[98] AJ Piergiovanni, Alan Wu, and Michael S Ryoo. Learning Real-World Robot Policies by Dreaming. In IROS, 2019. 21
[99] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. ICLR, 2024. 6, 20
[100] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- Regh-Resolttion Image Synthesis with Latent Diffusion Models. CVPR, 2022. 21
[101] Tim Salimans and Jonathan Ho. Progressive Distillation for Fast Sampling of Diffusion Models. In ICLR,
2023. 20
[102] Eder Santana and George Hotz. Learning a Driving Simulator. arXiv preprint arXiv: 2016.
2, 5, 7,
[103] Axel Sauer, Frederic Bocsel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast Heso-teeoolution Image Synthesis with Latent Adversarial Diffusion Distillation. arXiv preprint arXiv:2403.12015, 2024. 20
[104] Ingmar Schubert, Jingwei Zhang, Jake Bruce, Sarah Bechtle, Emilio Parisotto, Martin Riedmiller, Jost Tobias Springenherg, Arunkumar Byravan, Leonard Hasenelever, and Nicolas Heess, A Generalist Dynamics Model for Control. arXiv preprint arXiv:2305. 10t.::223...01..... 2023.
[105] Max Schwarzer. Ankesh Anand, Rishab Goel, Devon Hjelm. Aaron Courville, and Philip Bachman. Eita-EEficiiet Reinforcement Learning with Self-Predictive Representations, In ICLR, 2021, 21
[106] Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kylc Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine. ViNT: A Foundation Model for Visual Navigation. In CoRL, 2023. 19
[107] Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. DriveL.:: Driving with Graph Visual Question Answering. arXiv preprint arXiv:2312. 141500 2023. 20, 21
[108] Uriel Singer, Adam Polyak. Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Vides Text-to-Vidco Generation without Video Data. ICLR, 2023. 21
[109] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In ICLR, 2021.
22
[110] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency Models. In ICML, 2023. 20
[111] Yang Song, Jascha Dick-iicksten,, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based Generative Modeling through Stochastic Differential Equations. In ICLR, 2021.
15

---

[112]
Pei Sun. Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott M. Euinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov, Scalability in Perception for Autonomous Driving: Waymo Open Datasct. In CVPR, 2020. 8, 9, 20, 25, 26
[113] Richard $ Sutton. The Quest for Common Model of the Intelligent Decision Maker. arXiv preprint arXiv:2202. 13252, 2022. 20, 21
[114] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. In NeurlPS, 2020. 21
[115] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards Accurate Generative Models of Video: A New Metric & Challenges. arXiv preprint arXiv:1812.01711 2018. 7
[116] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In NeurlPS, 2017. 5, 21
[117] Vikram Voleti, Alexia Jolicoeur- Martinea, and Chris Pal. MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation. In NeurlPS, 2022. 21
[118] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin. Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel Multi- View Synthesis and 3D Generation from Single Image using Latent Video Diffusion. arXiv preprint arXiv:2403. /2008, 2024, 22
[119] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating Vidcos with Scene Dynamics. In NeurlPS, 2016. 21
[120] Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation. In ICCV, 2023. 21
[121] Jiuniu Wang. Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. ModelScope Text-to-Vidco Technical Report. arXiv preprint arXiv:2308 06571, 2023. 21
[122] Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Vide- oFactory: Swap Attention in Spatiotemporal Diffusions for Text-to- Video Generation. arXiv preprint arXiv:2305.10877 2023.
[123] Xiang Wang, Shiwei Zhang, Hangjie Yuan, Zhiwu Qing. Biao Gong, Yingya Zhang, Yujun Shen, Changxin Gao, and Nong Sang. A Recipe for Scaling up Text-to-Viee Video Generation with Text-Free Videos. In CVPR, 2024.
[124] Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videol.CM: Video Latent Consistency Model. arXiv preprint arXiv:2312.091000 2023. 20
[125] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. DriveDreamer: Towards Real- World-Driven World Models for Autonomous Driving, arXiv preprint 200.02230..09777, 2023. 1, 3,5,
7, 21
[126] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang. Ceyuan Yang. Yinan He. Jiashuo Yu, Peiqing Yang, Yuwei Guo, Tianxing Wu, Chenyang Si. Yuming Jiang. Cunjian Chen, Chen Change Loy, Bo Dai. Dahua Lin, Yu Qiao, and Ziwei Liu. LAVIE: Ligh-Quality Video Generation with Cascaded Latent Diffusion Models. arXiv preprini arXiv:2309. 15100091111....0111011101111 2023. 8, 20
[127] Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving. In CVPR, 2024. 3, 5, 6, 7, 19, 20,
[128] Zhouxia Wang. Ziyang Yuan, Xintao Wang. Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. MotionCtri: A Unified and Flexible Motion Controller for Video Generation. arXiv preprint arXiv:2312.03641, 2023. 6, 20, 21
[129] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagicet Singh, Siddhesh Whan Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, al. Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting. In NeurlPS Datasets and Benchmarks, 2023. 20
[130] Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge, Xiaodong Cun, David Junhao Zhang, Jia-Wei Liu, Yuchao Gu, Rui Zhao, Weisi Lin, Wynne Hsu, Ying Shan, and Mike Zheng Shou. Towards A Better Metric for Text-to- Video Generation. arXiv preprint 200 2024. 8
[131] Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre- Training Contextualized World Models with Ine Wild Videos for Reinforcement Learning. In NewrIPS, 2023. 21
16

---

[132] Penghao Wu, Li Chen, Hongyang Li. Xiaosong Jia, Junchi Yan, and Yu Qiao. Policy Pre-training for Autonomous Driving via Self-supervised Geometric Modeling, In ICLR, 2023. 4
[133] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang. Tien-Tsin Wong, and Ying Shan. DynamiCrafter: Animating Open-Doman Images with Video Diffusion Priors. arXiv preprint vrXiv2330.12219 2023. 7. 22
[134] Wilson Yan, Yunzhi Zhang. Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video Generation using
VQ-VAE and Transformers. arXiv preprint arXiv:2104. 10157, 2021. 21
[135] Xu Yan, Haiming Zhang, Yingjie Cai, Jingming Guo, Weichao Qiu, Bin Gao, Kaiqiang Zhou, Yue Zhao, Huan Jin, Jiantao Gao, Zhen Li, Lihui Jiang. Wei Zhang, Hongbo Zhang, Dengxin Dai, and Bingbing Liu. Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities. arXáv preprint wrXiv:2401.0004 2024. 1
[136] Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, Jun Zhang, Andreas Geiger, Yu Qiao, and Hongyang Li. Generalized Predictive Model for Autonomous Driving. In CVPR, 2024. 2, 3,4, 4, 6, 7, 8, 20, 21, 23, 25, 26
[137] Mengjiao Yang, Yilun Du Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter
Abbeel. Learning Interactive Real-World Simulators. In ICLR, 2024. 19, 21
[138] Sherry Yang. Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, Dale Schuurmans. Video as the New Language for Real World Decision Making. arXiv preprint
1rXiv:2402.17133 2024. 20
[139] Shiyuan Yang. Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a- Video: Customized Video Generation with User-Directed Camera Movement and Object Motion. arXiv preprint arXiv:2402.033
2024, 21
[140] Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li. Visual Point Cloud Forecasting Enables Scalable Autonomous Driving. In CVPR, 2024. 21
[141] Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes. arXiv preprint arXiv:2305. 10430, 2023. 9,
[142] Alex Zhang, Khanh Nguyen, Jens Tuyls, Albert Lin, and Karthik Narasimhan. Language-Guin Guided World Models: A Model-based Approach to Al Control. arXiv preprint 240iv:2402.0169) 2024. 21
[143] Lunjun Zhang, Yuwen Xiong. Yang. Sergio Casas, Rui Hu, and Raquel Urtasun. Learning Unsuper- vised World Models for Autonomous Driving via Discrete Diffusion. In ICLR, 2024. 1,21
[144] Shiwei Zhang, Jiayu Wang. Yingya Zhang. Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou, 12VGen-XL:
Image-to- -nage-to-Video Synthesis via Cascaded Diffusion Models. arXiv preprint arXiv:2311.04145 2023, 4, 7, 20, 21, 22
[145] Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang. Xiaoyi Bao, and Xingang Wang. DriveDreamer-2: L.LM-Enhangee World Models for Diverse Driving Video Generation. arXiv preprint
0rXiv:2403.06845, 2024. 20,21 21
[146] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. UniPC: A Unified Predictor-Correee Framework for Fast Sampling of Diffusion Models. In NeurlPS, 2023. 20
[147] Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu, OccWorld: Learning 3D Occupancy World Model for Autonomous Driving. arXiv preprint arXiv:2311.1603
2023. 2,
[148] Wenzhao Zheng, Ruiqi Song, Xianda Guo, and Long Chen. GenAD: Generative End-to-End Autonomous Driving, arXiv preprini 202.v:2402.11502. 2024. 19
[149] Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied Understanding of Driving Scenarios. arXiv preprint arXiv:2403.0 04593, 2024, 20
17

---

Appendix
Discussions
19
B Related Work
20
B.1 World Models
20
B.2 Video Generation
Implementation Details
21
C.I Model
C.2 Dataset
21
C.3 Training
22
C.4 Sampling
22
C.5 Human Evaluation
22
C.6 Reward Estimation
23
C.7 Ablation Studies
23
D Additional Experiments
23
D.I Parameter Sensitivity of Reward Estimation
23
D.2 Reward Estimation for Commands
23
D.3 Action Independence Constraint
23
D.4 Triangular Guidance Scheme
24
D.5 LoRA Adaptation
24
D.6 Action Control Consistency
D.7 Human Evaluation with GenAD
25
Additional Visualizations
25
E.1
Generalization Ability
25
E.2
Long-Horizon Prediction
25
E.3
Action Controllability
25
E,4 Counterfactual Reasoning Ability
25
E.5
Human Evaluation Cases
26
Licence of Assets
26
18

---

A Discussions
To help a thorough understanding of this work, we discuss intuitive questions that might be raised.
Q1. Why is at least position, velocity, and acceleration required to predict coherent futures?
Position ensures the predicted future begins continuously with the current state. Velocity manifests how objects are moving, e.g., whether they are turning left or turning right. Acceleration represents how velocity changes over time, e.g., whether the surroundings are moving faster or moving slower. Without utilizing acceleration as cue, car overtaking the ego ego-vehicle may suddenly be passed by in the next autoregressive prediction step. These three priors provide essential cues to allow consistent future extension with respect to historical observation.
Q2. How is the form of the proposed reward function defined?
Unlike VIPER [28] and Diffusion Reward [60] that both make discrete predictions, our model predicts continuous latent. Therefore, our reward is estimated according to conditional variance rather than likeliikl entropy. In addition, measuring uncertainty with log-likelihood requires comparing the prediction to the ground truth. As we deploy the reward in any scenario, the approach of VIPER is infeasible for our objective. Note that reward calculation is meticulously designed to satisfy the Kolmogorov axioms, i.e. it is non-negative and the measure of the entire sample space is [o, 1).
Q3. Reward estimation efficiency compared to the detector-based method [127].
Though our reward estimation involves multi-round denoising, it will not spend more compute than the detector-based reward function defined in Drive-WM [127]. To be specific, Drive-WM obtains the rewards from the perception results. Given that the detectors [82, 84] take image sequences as inputs, Drive-WXM has to accomplish all steps of the denoising process before perception. Differently, our reward function estimates the reward with the uncertainty that originates from the world model itself without relying on other perception models. Therefore, the estimation of uncertainty does not require completing the generation process. It can be realized by only denoising each sample for few steps. In fact, as specified in Appendix C, the total computation required for reward estimation per situation (10 steps, 5 rounds) is no greater than that of generating the entire video (50 steps for our model) as Drive-Wi. does. Note that the computational cost for our reward estimation can be flexibly reduced to further improve its efficiency. As shown in Fig. 14, using 5 denoising steps (50% of the default computation) also yields satisfactory estimations of the reward.
Q4. Usage of the proposed reward function.
(1) As mentioned Sec. 4.3, the proposed reward function can potentially serve as an alternative metric of driving actions that mitigate the concerns in existing open-loop evaluation [18, 83, 141]. (2) As demonstrated in Fig. 10, better actions generally yield higher rewards with our reward function. Taking advantage of this property, there is great promise for reward function to be used critic module [76], which enables model predicire control by executing the optimal action that maximizes the estimated reward [27, 30, 35]. This procedure can be performed in conjunction with distribution-based planners [53, 148] that can make action proposals to reduce the searching space.
Q5. Any other potential applications of Vista?
(1) As a generalizable predictive model, Vista could be utilized as forward dynamics model [13, 26] simulate short-term dynamics and assist long-horizon planning tasks like visual navigation [106]. (2) It is also intriguing to utilize Vista as an implicit driving policy, which is spontaneously acquired through future prediction [1, 25]. After synthesizing the video plan, we can convert the resultant image trajectory to executable actions by non-causal inverse dynamics model [4, 65, 71], which can be efficiently learned from much fewer data compared with the imitation learning pipeline [2, 9]. In autonomous driving, the inverse dynamics model could be implemented with visual odometry [75]. (3) In collaboration with the reward function, also worth investigating Vista could facilitate model-based reinforcement learning by boosting the sampling efficiency in real- world scenarios [137].
19

---

Q6. Differences with GenAD [136].
These two works have fundamental differences in control versatility and prediction fidelity. First of all, Vista is a generalizable world model that can controlled by multi-modal action conditions. Although GenAD has also trained on nuScenes and the generalization of its action control has never been validated. In contrast, Vista integrates versatile action controllability that can generalize to new scenarios in zero-shot manner. Unlike GenAD that requires labeling OpenDV- OpenDV-YouTubb with commands and texts, our collaborative training strategy skillfully averts this labor that may incur accumulated noises and conflicts [136]. In addition, Vista (10 Hz, 576x 1024) operates at much higher frame rate and resolution, considerably beyond the capability of GenAD (2 Hz,
extension, its weights are fully finetuned
in both temporal and spatial axes. Different from GenAD, we also put forth several dedicated designs for high-fidelity prediction. We find that Vista, with lower model complexity, achieves much better FID and FVD scores than GenAD (see Table 2).
Q7. Limitations, failure cases, and possible solutions.
As one of the pioneering efforts, Vista still has few limitations that call for future works. (1) Since Vista predicts futures at an exceptional spatiotemporal resolution, it is inevitable be computationally expensive, particularly in downstream applications. Potential solutions may include faster sampling techniques [89, 146] and training- based distillations [91, 93, 101, 103, 110, 124]. (2) It is possible that the prediction may undergo apparent degradation in manghhorizon rollouts or drastic view shifts. Extra refinements on the prediction results [6, 49, 54, 99, 126, 144] could helpful. Speculatively, applying our recipe to more scalable architecture [54, 97] is also promising to address this limitation. (3) Similar to other controllable video generation methods [128], the chance of failure still persists in our action controls, especially for ambiguous intentions such as commands and goal points as Fig. 8 reveals. Incorporating more datasets with action annotations [11, 112, 129] for collaborative training could be beneficial. Using compositional classifier-free free guidance [8, 19, 32] to amplify the individual impact of action conditions may also help (at cost of increased inference compute). (4) Although training data is based on the largest public driving dataset [136], it is nowhere near the entirety of Internet driving data, thus leaving huge untapped potential to further expand the capabilities of Vista.
Q8. Why not expand the Vista framework surround-view generation?
It is true that supporting surround-view generation would further help driving. Existing works [127, 145] have explored the surround-view settings on nuScenes [10]. However, in this paper, we focus on the front-view setting for three main reasons: (1) The front view setting allows leveraging diverse data sources [54, 136]. Conversely, the distinctions in multi-view videos from various datasets, such as different numbers of cameras, hinder unified modeling and data scaling. (2) Models that focus on the front view can be seamlessly applied to different datasets without adaptation [107], broadening their applicability across datasets. (3) Though incomplete, the front view often contains most of the information vital for driving. As demonstrated in NAVSIM [23], using the front view alone results in only a 1.1% performance downtick in collision rate compared to using five surround-vies cameras.
Q9. Broader impact.
Despite the encouraging improvements, our work is by no means perfect when it comes to applications that involve dealing with highly complicated situations. As Vista is based on the diffusion framework, which introduces stochastic outcomes and non-negligible latencies, deploying it into autonomous vehicles directly could pose safety risks. While it is not a silver bullet yet, we expect that Vista will inspire the community further exploit the capabilities and applications of driving world models. As prototype for generalizable driving world models, we hope that Vista can stimulate the investigations in developing generalizable systems for autonomous driving and machine intelligence.
B
Related Work
B.1
World Models
Intelligent agents should able to make efficacious decisions even under unseen situations 57, 76, 113, 138, 149]. This requires fundamental knowledge of the world that generalizes to rare cases. As an
20

---

internal manifestation of such knowledge. a world model predicts plausible futures of the world given potential actions [9. 40, 69, 76, 96, 113. 137]. In principle, not only predicts how the environment will unfold over time, but also deduces the underlying physical dynamics and agentic behaviors. Such 43,
properties can be useful for representation learning [35, 45, 88, 105, 131], model-based reinforcement learning [39, 40, 42, 43, 94, 96, 98], as well model-predictive control [27, 30, 35, 41, 92, 104, 142]. Recent methods [37, 44, 80, 85, 86] also induce language- -basd world models from large language models, but are restricted in textual space and struggle with grounding on physics [26, 59].
Although world models have been extensively applied and made significant revolutions in simulated games [40, 42, 43] and indoor embodiment [72, 92, 120]. such investigations for autonomous driving still lag behind [127. 143]. Different from other tasks, world modeling for autonomous driving poses unique challenges, which primarily arise from the large field of views with highly dynamic motions. Some practices imagine the world in the bird's eye view (BEV) space [20, 31, 52, 53, 78, 81]. Recent practices model the world state as raw sensor observations such point clouds [7, 67, 140, 143, 147] and images [54, 61, 68, 90, 102, 125, 127, 131, 145]. The latter category, namely visual world models, hold more promise for scaling up due to sensor flexibility and data accessibility. Nevertheless, existing methods are restricted to a particular dataset [61. 77, 90, 125, 127, 143, 145, 147] or simulator [7, 131], compromising their generalization ability to novel domains. Meanwhile, these efforts lack systematic designs for the driving domain and only model the world at relatively low frame rates and resolutions, which discards the fine aine-rrined details and impairs their ability express real-world behaviors. Moreover, most of them are restricted to a specific control modality [54, 61, 90, 125], which hinders the accommodation to prevailing planning algorithms [12, 14, 21, 56, 58, 64] and extension to more applications like carision-making [127] or user interaction [74]. Besides, existing methods seldom explore zero-shot action controllability across different datasets. The inferior generalization, fidelity and controllability collectively preclude existing driving world models from broadly facilitating the
development of autonomous driving.
B.2 Video Generation
Video generation is an effective way to model the world and has undergone remarkable advancements over the years. Pioncering works [119, 134] have studied various kinds of generative models [29, 33, 70]. Swayed by the success of diffusion models [24, 48, 100], a surge of liffusion-based video generation methods have emerged [6, 34, 46, 51, 108, 117, 121]. Recent works [5, 15, 32, 144] shift their focus towards image-to-videe in training data. However, most of them are not strict predictive models that generate videos starting from the condition image. Moreover, existing methods struggle with the intricate dynamics in driving scenarios from the ego perspective [136], which limits their feasibility as driving world models.
generation for its finer content description and better scalability
While the majority of existing methods produce videos without explicit controllability, two recent works [128, 139] introduce camera motion control to video generation. However, camera motion is conceptually distinct from vehicle actions and both of these works are text-to-video methods without any prediction ability. Contrarily, the model we developed is predictive world model that produces realistic dynamics and allows versatile action controls for autonomous driving.
C Implementation Details
C.1
Model
We adopt the framework of SVD [5] as the architecture of Vista, which consists of 2.5B parameters in total, including 1.6B UNet parameters. For action conditioning, we encode the value of each action sequence into Fourier embeddings [1 14, 116] with 128 channels.
C.2 Dataset
YouTube [136] for training, and incorporate nuScenes We utilize a rigorously filtered set of YouuTey training set [10] during the action control learning phase. Concretely, we manually eliminate 15 hours of irrelevant content from OpenDV-YouTube,, yielding approximately 1735 hours of unlabeled driving videos. Since nuScenes is heavily biased [83, 107, 127], we balance its samples based on command categories to foster the learning of rare actions. The video clips are sampled with 25 frames at 10 Hz. Although nuScenes [10] is logged at 12 Hz, we find negative impact of treating them
21

---

as 10 Hz videos. The model inputs are composed by cropping and resizing these clips to the target resolution. We define an action as a sequence comprising 25 frames. To categorize actions into commands, we follow the established conventions in planning [56, 58, 64] and define the command of ego-vehicle as "turn right" or "turn left" when its final displacement exceeds meters in the orthogonal direction relative to its initial heading. To allow more precise categorization, we additionally introduce a "stop" command when the forward driving distance is less than 2 meters.
C,3 Training
At the first training phase, we train all UNet parameters at 576x 1024 resolution on 128 A100 GPUs for 20K iterations, which takes about 8 days in total. We accumulate the gradients of 2 steps, yielding an effective batch size of 256. Following SVD, our model is trained with the EDM framework [66]. We use the AdamW optimizer [87] with a learning rate of 1 x 10-5. The learning rate for spatial layers is moderated by a discount factor of 0.1. The coefficients X1 and 22 in Eq. (6) are set to 1.0 and 0.1 respectively. Offset noise [38] is also used with strength of 0.02 as it helps improve temporal smoothness. We randomly sample different orders of dynamic priors with increasing probabilities, L.e. 1/15, 2/15, 4/15, 8/15 for 0. 1, 2, 3 condition frames respectively. The noise augmentation [49] is disabled to retain more details from the condition frames.
As for the action control learning phase, we freeze the pretrained weights and add LoRA [55] and projection layers all attention blocks of the UNet. The rank of LoRA set to 16. We then train the new weights at 576 resolution for 120K iterations using batch size 8 and learning rate 5 x 10- After the controllability can be clearly witnessed, we continue to finetune the unfrozen weights at 576x 1024 resolution for another 10K iterations. We drop out each activated action mode with a ratio of 15% to allow classifier-free guidance [50]. The sampling ratio of OpenDV-YouTube YouTube and nuScenes is 1 at this training phase. The whole training process for action controllability takes around 10 days on 8 A100 GPUs, with roughly 8 days the low resolution and days at the high resolution.
C.4 Sampling
We generate future videos using the DDIM sampler [109] for 50 steps. The sampling starts with max at 700.0. Since our model predicts long-term futures in an autoregressive manner, the issue of over-saturation caused by the standard classifier-free guidance will accumulate rapidly. Therefore. unlike SVD that linearly increases the guidance scale, we employ triangular classifier-free guidance scheme [118] to permit genuine long-horizon rollouts. Specifically, for the i-th frame in each K frames to predict, we assign its guidance scale s(i) following:
21 (8 max 8minn K 21K (i)) (Smax
if K
(1)
8min
mmax
Smin)
if 2
K 2
where Smin and Smax indicate the minimum and maximum guidance scales along the temporal axis. In our experiments, we define Smin as 1.0 and smax as 2.5. This triangle scheme assigns moderate guidance scales to the frames that will be used as conditions in the next prediction round. Due to sufficient temporal interaction, the quality of intermediate frames can also propagate to the frames that have lower guidance scales. As illustrated in Fig. 15, this technique adeptly mitigates the saturation drift problem while enhancing details. To improve perceptual continuity, we split the generated latent into clips with an overlap of 3 frames before sending them to the video-aware decoder [5]. After decoding, the overlapped frames are averaged pixel-wise.
C.5 Human Evaluation
Recall that we ask the participants to judge side-by-side video pairs from visual quality and motion rationality. To guarantee credible responses, we provide detailed commentary for each aspect of the human evaluation. For visual quality, we let the participants focus on the consistency and harmony of the generated content. For motion rationality, we encourage the participants to pay more attention to the plausibility of the ways that ego-vehicle and other agents move, e.g., whether they are following the traffic rules and exhibiting safe behaviors. For all public models we compared, we use the official checkpoints and configurations for inference without finetuning. For the models that require textual inputs [133, 144], we set the prompt as "realistic drive view".
22

---

Table 5: Effect of action independence. Without losing generality, we choose the trajectory representative action for evaluation. The proposed
Table 4: Reward commands on Waymo. The ground truth commands generally obtain higher rewards than random command inputs, suggesting that the proposed reward function can be used as a reliable indicator for command selection.
constraint expedites the learning of actions.
Subset FVD.
Strategy
Action
forth
right
left
stop
Condition
Average Reward
w/o Traj.
163.0
273.9
428.3
497.1
w/o A.I.
GT Com.
0.892 0.878 (-0.014)
w/ Traj-
138.8
232.9
368.2
132.3
random Com.
who Traj.
156.2
263.7
402.9
463.7
w/ A.I.
w/ Traj.
130.7
230.8
345.7
118.9
C.6 Reward Estimation
For each condition frame and action pair, we accumulate ensemble with size M 5 to obtain reliable uncertainty estimation. Each sample in the ensemble is inferred for 10 denoising steps as we find it is unnecessary to generate high-quality results for uncertainty estimation. The coefficient / in the correlating strategy [35, 95] is set to 0.5.
C.7 Ablation Studies
For the ablation of loss functions, we train each variant OpenDV-YouTube a spatial resolution 576x 1024. All ablations, including the additional ablations in Appendix D, initialized by loading the pretrained checkpoint of SVD [5] and conducted with 8 A100 GPUs.
YouTube [136] for 10K steps
D Additional Experiments
D.1 Parameter Sensitivity of Reward Estimation
denaie ensemble doncke step. ansemble siee
Average Reward
To investigate how the number of denoising steps and the en- semble size influence the performance of the proposed reward function, we repeat the reward estimation procedure with different hyperparameter settings. We start off by using 5 de- noising steps and an ensemble size 5 for each sample. We then test two variants that double the computational cost by increasing the denoising steps to 10 (our default setup in Appendix C) and in- creasing the ensemble size to 10 respectively. Following Sec. 4.3, we plot the correlation of the estimated rewards with L2 errors for the three variants in Fig. 14. The results show that increasing the number of denoising steps can greatly enlarge the relative contrast of rewards, suggesting that denoising step is a more important factor than ensemble size under the same computation budget for reward estimation.
Sec, 4.3
6:50
12 Error
Figure 14: Sensitivity of reward estimation hyperparameters. Increasing the number of denois- ing steps can produce more dis- criminative rewards, whereas in- creasing the ensemble size can slightly stabilize the estimations.
D.2 Reward Estimation for Commands
To demonstrate that the proposed reward function also applicable for other actions, we estimate the rewards of ground truth commands from Waymo and compare them with the rewards of random commands. The results in Table 4 suggest that our reward is also competent for command selection.
D.3 Action Independence Constraint
To prove the efficacy of our learning strategy for action control, we conduct comparison by removing the action independence constraint proposed in Sec. 3.2. We train two variants on nuScenes [10] at the resolution of 576 pixels for 62K steps. The comparison results presented in Table 5.
23

---

none constant
Inear (SVD) triongular (Ours)
Figure 15: Effect of guidance scale. We predict 15s long-term videos with different CFG schemes.
Our method achieves the optimal equilibrium between detail generation and saturation maintenance.
turn left
start frome
turn right
wlo
Figure 16: Necessity of LoRA adaptation. Training newly added projections alone without LoRA results in visual corruptions. The compared variants are trained on nuScenes and inferred on Waymo.
D.4 Triangular Guidance Scheme
We further compare the introduced classifier-free guidance scheme with the vanilla scheme and the linear scheme [5] verify its necessity. Fig. 15 shows that our triangular scaling attains the best trade-off between visual quality and saturation preservation.
D.5 LoRA Adaptation
To show the necessity of applying LoRA in Sec. 3.2, we train two variants resolution of 320 pixels for 30K iterations. With the pretrained UNet weights fixed, we let one variant train LoRA and action projection layers in the attention blocks, while the other adjusts new projection layers only. shown in Fig. 16, adding LoRA is essential for action control learning.
D.6 Action Control Consistency
In Table 6, we report the complete FVD scores of Fig. 8, which further validates the effectiveness of all kinds of action controls. Note that since our "stop" subset consists of samples where the final displacements are within 2 meters, the goal points typically do not appear in these videos. Hence, for the experiment that uses goal point as action condition on the "stop" subset, most samples are generated in the same way the action-free mode.

---

Table 6: Complete FVD scores of different action categories. We obtain the FVD scores four
subsets divided by command categories. All types of action controls are effective across all categories.
Subset FVD left
Dataset
Condition
forth
right
stop
average
action-fice
135.6
405.6
513.8
414.1
367.2
+ goal point
122.4
315.6
439.6
413.5
322.7
nuScenes
+ command angle speed
122.2
299.7
485.6
261.6
292.2
122.8
285.6
397.8
114.1
230.0
+ trajectory
125.2
229.2
357.7
118.5
207.6
action-frec
145.9
407.6
529.9
164.1
311.8
Waymo
command
122.5
331.5
496.9
143.9
273.7
trujectory
126.3
285.5
527.6
136.5
268.9
D.7
Human Evaluation with GenAD
To our best knowledge, there is not a driving-specific world model publicly available so far, making hard conduct qualitative human evaluation. Hence, we mainly compare Vista against the existing methods with the officially reported FID and FVD scores in Table 2.
To demonstrate the considerable improvements in visual quality and motion rationality, we conduct an extra human evaluation with the state-of- the-art oftte-of-the-ar GenAD model [136]. Since GenAD processes a 4-second video each time, we perform autoregressive prediction extend Vista 's output to 5 seconds and trim the last second to align with GenAD's s duration. To avoid any bias caused by resolution and frequency, we downsample the outputs of Vista (576x 1024 resolution at 10 Hz) to 256x448 resolution at 2 Hz. The evaluation follows the same procedure specified in Sec. 4.1.
storfing from
Dis CanAD
1234
44
We collect 25 diverse samples from the unseen OpenDV- OpenDV-YouTube Yubeube- val set and invite 20 volunteers for evaluation. We ask the volun- teers to choose the video they deem better. As a result, Vista is preferred in 94.4% and 94.8% of the time visual quality and motion rationality respectively. This attests that Vista, spite of experiencing large perceptual reduction due downsam- pling, still exhibits a significant ad advantage over GenAD in generation quality. We also compare the predictions of GenAD and Vista in Fig. 17, showing the superiority of Vista in resolution and fidelity.
Figure 17: Perceptual disparity between GenAD and Vista.
E
Additional Visualizations
E.1
Generalization Ability
We further demonstrate the strong generalization ability of Vista by deploying it to different scenarios in the wild. The results in Fig. 18 and Fig. 19 illustrate that Vista make high-fidelity predictions in very diverse range of scenarios.
E.2 Long-Horizon Prediction
In addition to Fig. 6, we provide more qualitative visualizations of oog-horizoo prediction in Fig. 20. Vista can continuously predict long-term futures with consistent content and motion.
E.3 Action Controllability
We provide more prediction results with different action inputs in Fig. 21. The results on OpenDV- YouTube-val [136] and Waymo [112] show that the versatile controllability of Vista be readily transferred to different domains in a zero-shot manner.
E.4 Counterfactual Reasoning Ability
Counterfactual reasoning ability is one of the emergent abilities of world models [36]. As shown in Fig, 22, Vista can effectively predict the counterfactual consequences caused by abnormal actions.
25

---

E,5
Human Evaluation Cases
To demonstrate the diversity of the scenes selected for human evaluation (Sec. 4), we show all cases gathered from OpenDV- YouTube-val [136], nuScenes [10]. Waymo [112], and CODA [79] in Fig. 23.
F Licence of Assets
Our training and evaluation utilize the data from four publicly licensed datasets [10, 79, 112, 136). Our implementation based on the codebase SVD [5], which uses the MIT license. The pretrained checkpoint of SVD is distributed under the stable video diffusion non-commercial community license.
26

---

Figure 18: Generalization ability of Vista. We apply Vista across diverse scenes (e.g., countrysides and tunnels) with unseen camera poses (e.g., the perspective of double-decker bus). Our model can predict high-resolution futures with vivid behaviors of vehicles and pedestrians, exhibiting strong generalization abilities and profound comprehension of world knowledge. Best viewed zoomed in.
27

---

Figure 19: Generalization ability of Vista in more scenarios. Best viewed zoomed in.
28

---

15c
Figure 20: Additional results of long-horizon prediction. Our model can autoregressively simulate
long driving experiences with marginal quality decline. All videos continue for 15 seconds 10 Hz.
29

---

trjjectory
"slap"
"uuniggh
speed
"umm left
'1ep'
"tum
point
Figure 21: Additional results of action controllability. We trial different action conditions across multiple scenes from CounDV-YouTube-ooe. and Waymo. The behaviors of the ego-vehicle can be consistently controlled by various kinds of interventions.
30

---

ground
mulh
abnorma
oction
ground
truth
obnormal oction
Figure 22: Counterfactual reasoning ability. By imposing actions that violate the traffic rules, we discover that Vista can also predict the consequences of abnormal interventions. In the first example, the eeo-vehicle passes over the road boundary and rushes into the bush following our instructions. In the second example, the passing car stops and waits to avoid a collision when force the ego-vehicle to proceed at the crossroads. This showcases Vista's 's potential for facilitating closed-loop simulation.
Figure 23: Diverse scenes collected for human evaluation. We carefully curate 60 scenes from OpenDV-Youle YouTubebulbllll nuScenes, Waymo and CODA. The distinctive attributes each dataset jointly represent the diversity of real-world environments, permitting comprehensive human evaluation.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately summarize the contributions and reflect our results in Sec. 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See the conclusion and discussions Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
32

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See the implementation details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification: The code for Vista has been attached to the supplementary material. It will be released public later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
httpc//giidee/oeeeuunnttteee soioooicsin for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSubmiis
httpps.ccppubiic//eecc / ssi asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
See the experimental details Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experiments are conducted at an extensive data scale as specified in Sec. 4. We also provide detailed studies about hyperparameter sensitivity in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the eтror bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
34

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See the implementation details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: p/tpp//ccccccppppipiii////////// httpethhiiccieeeeeeeeeeeeee
Answer: [Yes]
Justification:
The authors have reviewed and understood the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See our discussions in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix F.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code will be released with well-organized documentation and instructions.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
See our settings in Sec. 4.1 and Appendix C.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper has no potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "TLPksz8qSB", "reviewer": "Reviewer_Fp7V", "summary": "The paper presents a method named Vista, a novel driving world model that addresses limitations in generalization, prediction fidelity, and action controllability. Key contributions include introducing novel loss functions for high-resolution prediction, a latent replacement approach for coherent long-term rollouts, and versatile action controls ranging from high-level commands to low-level maneuvers. Vista demonstrates strong generalization in zero-shot settings and establishes a generalizable reward function for evaluating driving actions. Extensive experiments on multiple datasets demonstrate that Vista outperforms advanced general-purpose video generators in over 70% of comparisons, surpasses the best-performing driving world model by 55% in FID and 27% in FVD, and establishes a generalizable reward function for real-world driving action evaluation.", "questions": "1. Can the authors provide more details on the implementation of the dynamic prior injection mechanism? How does it theoretically support coherent long-term rollouts?\n2. Provide additional experimental results demonstrating the model's performance in zero-shot settings across diverse and unseen environments. This would help validate the framework’s generalization capabilities and robustness in real-world applications.\n3. Expanding the Vista framework to support surround-view video generation could significantly enhance its applicability and robustness in real-world autonomous driving scenarios, as a comprehensive 360-degree view is crucial for effective navigation and situational awareness. Additionally, extending the closed-loop evaluation to include a broader range of scenarios and detailed performance metrics would provide a more thorough assessment of the model's robustness and reliability. Such enhancements would further solidify the contributions of this work and pave the way for future advancements. Nonetheless, the current contributions are commendable, and I look forward to seeing future developments in this area.", "limitations": "please refer to weaknesses and questions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper introduces a novel approach to driving world models by incorporating high-fidelity predictions and versatile action controllability. This combination addresses existing gaps in generalization, prediction fidelity, and action flexibility, representing a significant step forward in autonomous driving research. \n2. The paper presents a well-designed methodology that systematically addresses the limitations of existing driving world models. The integration of dynamic prior injection and versatile control mechanisms is methodologically sound and effectively implemented.\n3. The paper is well-structured, with clear and logical sections that guide the reader through the problem formulation, methodology, experiments, and conclusions. Figures and tables are used effectively to illustrate key points and results.", "weaknesses": "1. Vista seems cannot to generate surround-view video, this limitation may restrict the method's effectiveness and generalizability in real-world scenarios where a comprehensive 360-degree view is crucial.\n2. The closed-loop evaluation is demonstrated through only a few cases, raising concerns about the robustness and reliability of integrating the closed-loop process with the generative model. Expanding the evaluation to include a broader range of scenarios and detailed performance metrics would help assess the seamless integration of the closed-loop driving process with the generative model.\n3. The paper introduces dynamic prior injection for coherent long-term rollouts, but the implementation details and impact of this component are not sufficiently elaborated. Providing more detailed information on this mechanism, including its implementation, theoretical basis, and specific impact on long-term prediction consistency, would enhance understanding.\n4. The paper lacks a detailed analysis of the computational resources required for training and inference using the Vista framework. The potentially high computational demands could limit the scalability and real-time applicability of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful and positive feedback. The following are our responses.\n\n> **W1&Q3**: Surround-view generation is not supported.\n\nWe agree that supporting surround-view generation would further help driving. We are planning to extend Vista to multi-view settings like Drive-WM (Wang, et al.) in our future works.\n\nIn this paper, we focus on the front-view setting for three main reasons:\n- The front view setting allows leveraging diverse data sources (e.g., the worldwide OpenDV dataset). Conversely, the distinctions in multi-view videos from various datasets, such as different numbers of cameras, hinder unified modeling and data scaling.\n- Models that focus on the front view can be seamlessly applied to different datasets without adaptation (e.g., DriveLM (Sima, et al.)), broadening their applicability across datasets.\n- Though incomplete, the front view often contains most of the information necessary for driving. As demonstrated in NAVSIM (Dauner, et al.), using the front view alone results in only a 1.1% performance drop in collision rate compared to using five surround-view cameras.\n\n> **W2&Q3**: Evaluating the closed-loop process in a broader range of scenarios and detailed metrics.\n\nFrom our understanding, the closed-loop process here refers to controlling Vista to create a closed-loop simulation. \n\nTo address the concern, we introduce an additional metric, *Trajectory Difference*, to assess the control consistency. Following GenAD, we train an inverse dynamics model (IDM) that estimates the corresponding trajectory from a video. We then send Vista's prediction to the IDM and calculate the L2 difference between the ground truth trajectory and the estimated trajectory. The lower the difference, the better the control consistency Vista has. We conduct the experiments on nuScenes and Waymo (unseen by Vista). As reported in **Table 1 of the rebuttal PDF**, Vista can be effectively controlled by different types of actions, yielding more consistent motions to the ground truth.\n\nWe also provide the complete FVD scores of Fig. 7 in **Table 2 of the rebuttal PDF**, which further validates the efficacy of all kinds of action controls.\n\nFor the coverage of scenarios, we have shown multiple open-world samples on the anonymous demo page at the time of submission. We will provide more demonstrations in the revision. In addition, we will fully open-source our code and model to the community for free trials.\n\n> **W3&Q1**: More details on the implementation, theoretical basis, and impact of dynamic prior injection.\n\n**Implementation details**: Conventional video diffusion methods (e.g., SVD) process a sequence of noisy frames to generate a video. However, this approach cannot ensure content and motion consistency between clips in long-horizon rollouts. To address this, we inject previous frames as priors to derive the necessary information for consistent rollouts. As illustrated in Fig. 2 [Left], these dynamic priors are injected by replacing the noisy frames with the previously known frames throughout the entire denoising process. The dynamic priors are then propagated through the temporal interactions within the model. The number of dynamic priors corresponds to the number of the injected frames. To indicate the presence of dynamic priors that do not require denoising, we assign different timestep embeddings to these frames. In our implementation, we create a frame-wise mask to uniformly allocate the dynamic priors and the timestep embeddings. We will refine this part accordingly in the revision.\n\n**Theoretical basis**: As discussed in Appendix A-Q1, it is necessary to input sufficient information so that the model can learn to derive position, velocity, and acceleration for coherent future prediction. For example, without knowing acceleration, the model cannot determine whether another car in view is moving faster or slower. Such uncertainty will result in unnatural motions with respect to the historical frames. To fully obtain these priors, at least three consecutive frames with the same interval are required. To ensure temporal consistency while predicting as many frames as possible each time, we always use three previous frames as dynamic priors during long-horizon rollouts.\n\n**Impact evaluation**: To further demonstrate the effectiveness of dynamic priors, we conduct a quantitative evaluation in **Table 1 of the rebuttal PDF**. Specifically, we use the inverse dynamics model to infer the trajectories of the predicted videos with different orders of dynamic priors. Extensive results show that increasing the order of dynamic priors can consistently improve the coherence to the ground truth motion.\n\n> **W4**: A detailed analysis of the computational resources required for training and inference.\n\nWe have provided a thorough description of the training in Appendix C.3. The entire training process takes about two weeks, with the first phase taking one week on 128 A100 GPUs and the second phase taking another week on 8 A100 GPUs. For inference cost, it takes 70-80 seconds on a single A100 to predict 25 frames. Note that the inference could be greatly accelerated using some well-established techniques as discussed in Appendix A-Q7. While this is not in the scope of this paper, we will explore these techniques for downstream applications in the future.\n\n> **Q2**: Additional experimental results in zero-shot settings across diverse and unseen environments.\n\nAll qualitative visualizations in our paper are produced under the zero-shot setting in open-world scenarios. Moreover, except for the results on nuScenes, all quantitative results are demonstrating Vista’s zero-shot performance (quality, duration, controllability, etc.). As described in Sec. 4.1, our experiments involve zero-shot evaluation on three unseen/geofenced datasets (OpenDV-YouTube-val, Waymo, CODA). Fig. 20 also shows the environmental diversity of our human evaluation. We will specify the zero-shot settings in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bQQicO7OVU", "reviewer": "Reviewer_ApTu", "summary": "Vista is a generalizable driving world model that excels in high fidelity and versatile controllability. By introducing novel losses to enhance the learning of dynamics and structural information, and integrating a unified conditioning interface for diverse action controls, Vista achieves high-resolution predictions and adapts seamlessly to various scenarios in a zero-shot manner. Extensive experiments demonstrate that Vista outperforms state-of-the-art video generators and driving models, showcasing significant improvements in prediction fidelity and action evaluation capabilities​.", "questions": "Mentioned in the weakness section", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. High Fidelity Predictions: Vista achieves accurate and realistic future predictions at high resolutions.\n2. Versatile Action Controllability: Supports diverse control inputs from high-level commands to low-level maneuvers.\n3. Strong Generalization: Seamlessly adapts to diverse and unseen environments in a zero-shot manner.\n4. Superior Performance: Outperforms state-of-the-art models in prediction fidelity and evaluation metrics.", "weaknesses": "1. I have some questions regarding the production of Figure 5. Is the SVD in Figure 5 used as is, or has it been retrained? Are there any other action control modules? Additionally, while the long-term generation of Vista seems consistent, most of the details are quite blurry. I am also confused about the point that \"the prediction of SVD does not commence from the condition image.\"\n\n2. Regarding the misalignment in DynamiCrafter in Figure 4, it is actually because during training of DynamiCrafter , the condition frame is not always the first frame, but is randomly extracted from the video. Therefore, you can see that the fourth column of DynamiCrafter is consistent with the input frame. This is not a misalignment.\n\nI will raise my rating if all of my concerns are well addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful comments and questions. We answer each question below and will incorporate all feedback in the revision.\n\n> **W1**: Details related to the production of Fig. 5. (1) Has SVD been retrained? Are there any other action control modules? (2) The details of long-term generation are blurry. (3) The meaning of \"the prediction of SVD does not commence from the condition image\".\n\n(1) We did not retrain SVD for this comparison. The results of SVD in Fig. 5 are generated using the official checkpoint and codebase without any modification. The samples in Fig. 5 are all action-free predictions without action conditioning, thus no action control modules are needed for SVD here.\n\n(2) It is possible that the long-horizon rollouts may result in degradations of details. In fact, long-term prediction remains a challenge in this research direction. To overcome this challenge, we have explored some techniques that optimize the fidelity of long-term prediction (e.g., triangular classifier-free guidance in Appendix C.4). Moreover, although there is still room for improvement, it is noteworthy that our method has significantly outperformed the existing methods, with nine times more frames than Drive-WM and much better content consistency compared to SVD. As discussed in Appendix A-Q7, we will continue exploring solutions for long-term fidelity in future works.\n\n(3) We apologize for the confusion. We use \"the prediction of SVD does not commence from the condition image\" to express that the first frame predicted by SVD is not identical to the condition image. This misalignment prevents SVD from performing autoregressive long-term rollout, as the consecutive clips predicted by SVD are not consistent in content. We will clarify this in the revision.\n\n> **W2**: The misalignment in DynamiCrafter is because of its training settings.\n\nThanks for the comment. We agree that the misalignment of DynamiCrafter is due to its training settings, where a random frame is sampled as the condition image. Since DynamiCrafter is a prominent general-purpose video generator, we compare our model to DynamiCrafter to demonstrate the better fidelity of Vista and its distinctions in capabilities. Unlike existing general-purpose video generators for content creation, Vista is designed to make plausible future predictions while preserving high quality. We will clarify this in our revision to avoid confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8QFkjuqaS0", "reviewer": "Reviewer_bcia", "summary": "This paper presents Vista, a driving world model that predict future driving video based on video diffusion model. In particular, the author introduces the idea of conditioning on prior frames and two domain-specific losses to capture dynamics and preserve structures for driving scenarios. By using LoRA, Vista also supports controllable (via e.g., trajectory, angle/speed) generation in a zero-shot manner. The author evaluates Vista by showing its high-resoluation videos, low FID on nuScenes and high human evaluation scores.", "questions": "-Can you provide more discussions on why the proposed methods are very novel when there are already similar work (e.g., GenAD) on this similar task?\n\n-Potential human evaluation results between Vista and GenAD (and/or other driving domain specific generation models)?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-neat domain-specific improvement on controllable driving video generation.\n\n-the paper is clear and easy to follow.", "weaknesses": "-while the author discussed the differences between Vista and GenAD in Appendix A-Q6 (e.g., different controllability design, and results with higher resolution and lower FID), the improvements seem to be on the incremental side. Besides, GenAD additionally supports text command and its qualitative results of GenAD looks as good as Vista.\n\n-the proposed conditioning on the prior frame and the two domain-specific losses and the use of LoRA while being helpful and neat, are not super novel. \n\n-human evaluation was not conducted between Vista and driving specific generation models (e.g., GenAD).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and effort in reviewing our paper. We provide detailed explanations below to solve the questions and some potential misunderstandings.\n\n> **W1&W2&Q1**: The improvements from GenAD seem to be incremental. The proposed dynamic priors, loss functions, and LoRA adaptation are not super novel when there are already similar works (e.g., GenAD) on this similar task.\n\nIn our humble opinion, there are several fundamental differences between Vista and previous works like GenAD. Our work is a pioneering attempt to build a generalizable high-fidelity driving world model that can be controlled by multi-modal actions. To this end, we propose a series of improvements that effectively extend Vista’s abilities:\n- With our efficient training strategies, Vista has acquired versatile action controllability that can be readily generalized to unseen scenarios. In stark contrast, none of the existing works has ever enabled such ability.\n- We propose a novel reward function and validate its zero-shot efficacy in Fig. 9 and Table 3, which could serve as a potential avenue to assess actions in the wild.\n- Thanks to our innovative techniques, Vista achieves a non-marginal performance gain of **55%** in FID and **51%** in FVD compared to GenAD while being more compact in model size. We provide an intuitive comparison between Vista and GenAD in **Fig. 2 of the rebuttal PDF**. We also meticulously conduct a human evaluation between these two methods, where Vista achieves a **94.4%** win rate in Visual Quality and a **94.8%** win rate in Motion Rationality. Please refer to our answer to the next question.\n- Beyond the superior visual quality and spatiotemporal resolution, we also enable coherent long-horizon prediction ability, which is underexplored in previous works.\n\nWe believe that all these contributions will shed light on future investigations in developing driving world models.\n\n> **W1&W3&Q2**: Human evaluation was not conducted between Vista and driving-specific generation models (e.g., GenAD). Qualitative results of GenAD look as good as Vista.\n\nTo our best knowledge, there is not a driving-specific world model publicly available so far, making it hard to conduct qualitative human evaluation. Therefore, we mainly compare Vista against the existing methods with the officially reported FID and FVD scores in our paper.\n\nTo demonstrate the considerable improvements in visual quality and motion rationality, we conduct a human evaluation with the state-of-the-art GenAD model *as requested*. We follow the two-stage training strategy and use the same training compute as specified in their original paper. Since GenAD processes a 4-second video each time, we perform autoregressive prediction to extend Vista’s output to 5 seconds and trim the last second to align with GenAD’s duration. To avoid any bias caused by resolution and frequency, we also downsample the outputs of Vista (576x1024 resolution at 10 Hz) to 256x448 resolution at 2 Hz for fairness. The human evaluation is conducted following the same procedure in Sec. 4.1.\n\nDuring the rebuttal period, we collect 25 diverse samples from the unseen OpenDV-YouTube-val set and invite 20 volunteers for evaluation. We ask the volunteers to choose the video they deemed better. As a result, Vista is preferred in **94.4%** and **94.8%** of the time on Visual Quality and Motion Rationality respectively. This certifies that Vista, even when downsampled to a much lower spatiotemporal resolution (which is a significant perceptual reduction), has a remarkable advantage over GenAD in generation quality. We also append a qualitative comparison between GenAD and Vista in **Fig. 2 of the rebuttal PDF**, showing the superiority of Vista in resolution and quality.\n\n> **W1**: Vista does not support text commands while GenAD does.\n\nAlthough GenAD provides a control interface for text commands, its advantage over Vista is limited for three main reasons:\n- The text commands used by GenAD are annotated by first classifying videos into discrete categories and then mapping to text templates from a predefined dictionary. Thus, the text commands used in GenAD and categorical embeddings learned by Vista are **functionally similar**.\n- As stated in the limitations of the GenAD paper, the auto-labeling process of text annotations may incur conflicts with ego intentions. Unlike GenAD that requires labeling all videos from YouTube, we explore a collaborative training strategy that allows learning open-world action controllability from the public dataset with reliable annotations. This strategy circumvents the labor of auto-labeling that may cause accumulated errors and learning instability.\n- Text commands are often ambiguous in expressing precise controls. Therefore, in Vista, we focus more on versatile action controllability ranging from high-level intentions to low-level maneuvers, empowering the applications for various purposes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and ACs:

We express our sincere gratitude to the reviewers for their thorough and constructive comments. It is encouraging that all reviewers have acknowledged our pioneering efforts in establishing a driving world model with versatile controllability.

We have carefully taken each comment into consideration. The attached **rebuttal PDF** includes two tables with quantitative results and two figures for illustration. Please refer to other rebuttal modules below for our detailed responses to each comment. We will integrate these results and discussions into our revised paper.

We hope that our rebuttal can address the concerns you may have. You are more than welcome to ask any further questions. We are looking forward to your feedback!

Best regards,  
Authors of Submission574

Meta-review: Overall, the reviewers have positively recommended the paper. The ACs agree with the reviewers' comments. 
Here are some additional comments for the authors to potentially improve the quality of the final camera-ready version:
	1	Quantitative evaluation could be more comprehensive -> they only show evaluation on nuScenes val set 
	2	No mention of the baseline models used for the human evaluation study — they tested on waymo and coda so which models were used for that? 
	3	Most of the videos/demos on the website are of empty streets. How good is the model in predicting motion of other vehicles/objects in the scene?
	4	For the new losses, we only see two examples that motivated the use of the new losses (Fig 12). That might not be very comprehensive. Maybe quantitative evaluation showing how much these losses affect the performance?
	5	They introduce different control techniques but they don’t use them simultaneously. It would be interesting to see an experiment with multiple controls used together. Would that be too strong of a signal for the generations? How could these controls interact together? 

Unanswered questions:
	1	Which resolution is used for the quantitative evaluation ? Are the metrics much better for Vista because of the resolution? 
	2	The controls are mainly used from NuScenes because it’s the only dataset with these annotations. Why not trying unsupervised/semi-supervised methods to get these annotations for more data? Would that affect the controllability ?
Decision: Accept (poster)