Title: Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
Venue: NeurIPS 2024
Accepted: yes
Keywords: World Model

Autonomous Driving

Video Prediction
Abstract: World models can foresee the outcomes of different actions, which is of paramount importance for autonomous driving. Nevertheless, existing driving world models still have limitations in generalization to unseen environments, prediction fidelity of critical details, and action controllability for flexible application. In this paper, we present Vista, a generalizable driving world model with high fidelity and versatile controllability. Based on a systematic diagnosis of existing methods, we introduce several key ingredients to address these limitations. To accurately predict real-world dynamics at high resolution, we propose two novel losses to promote the learning of moving instances and structural information. We also devise an effective latent replacement approach to inject historical frames as priors for coherent long-horizon rollouts. For action controllability, we incorporate a versatile set of controls from high-level intentions (command, goal point) to low-level maneuvers (trajectory, angle, and speed) through an efficient learning strategy. After large-scale training, the capabilities of Vista can seamlessly generalize to different scenarios. Extensive experiments on multiple datasets show that Vista outperforms the most advanced general-purpose video generator in over 70% of comparisons and surpasses the best-performing driving world model by 55% in FID and 27% in FVD. Moreover, for the first time, we utilize the capacity of Vista itself to establish a generalizable reward for real-world action evaluation without accessing the ground truth actions.

Vista : A Generalizable Driving World Model with High Fidelity and Versatile Controllability

Abstract

World models can foresee the outcomes of different actions, which is of paramount importance for autonomous driving. Nevertheless, existing driving world models still have limitations in generalization to unseen environments, prediction fidelity of critical details, and action controllability for flexible application. In this paper, we present Vista , a generalizable driving world model with high fidelity and versatile controllability. Based on a systematic diagnosis of existing methods, we introduce several key ingredients to address these limitations. To accurately predict real-world dynamics at high resolution, we propose two novel losses to promote the learning of moving instances and structural information. We also devise an effective latent replacement approach to inject historical frames as priors for coherent long-horizon rollouts. For action controllability, we incorporate a versatile set of controls from high-level intentions (command, goal point) to low-level maneuvers (trajectory, angle, and speed) through an efficient learning strategy. After large-scale training, the capabilities of Vista can seamlessly generalize to different scenarios. Extensive experiments on multiple datasets show that Vista outperforms the most advanced general-purpose video generator in over 70% of comparisons and surpasses the best-performing driving world model by 55% in FID and 27% in FVD. Moreover, for the first time, we utilize the capacity of Vista itself to establish a generalizable reward for real-world action evaluation without accessing the ground truth actions.

1 Introduction

Driven by scalable learning techniques, autonomous driving has made encouraging strides over the past few years [18, 58, 135]. However, intricate and out-of-distribution situations are still intractable for state-of-the-art techniques [83]. One promising solution lies in world models [57, 76], which infer the possible future states of the world from historical observations and alternative actions, in turn assessing the feasibility of such actions. They hold the potential to reason with uncertainty and avoid catastrophic errors [54, 76, 127], thereby promoting generalization and safety in autonomous driving.

Although a primary prospect of world models is to enable the generalization ability to novel environments, existing driving world models are still constrained by data scale [90, 125, 127, 143, 147] and geographical coverage [54, 61]. As summarized in Table 1 and Fig. 1, they are also often confined to low frame rates and resolutions, resulting in a loss of critical details. Furthermore, most models only support a single control modality such as the steering angle and speed. This is insufficient to express various action formats ranging from high-level intentions to low-level maneuvers, and incompatible with the outputs of prevalent planning algorithms [12, 14, 21, 56, 58, 64]. In addition, generalizing action controllability to unseen datasets is understudied. These limitations impede the applicability of existing works, making it imperative to develop a world model that overcomes these limitations.

Method

Resolution

Angle&Speed

Trajectory

Command

DriveSim [102]

7h

5 Hz

80

×

160

✓

DriveGAN [68]

160h

8 Hz

256

×

256

✓

DriveDreamer [125]

5h

12 Hz

128

×

192

✓

Drive-WM [127]

5h

2 Hz

192

×

384

✓

WoVoGen [90]

5h

2 Hz

256

×

448

✓

ADriver-I [61]

300h

2 Hz

256

×

512

✓

GenAD [136]

2000h

2 Hz

256

×

448

✓

✓

GAIA-1 [54]

4700h

25 Hz

288

×

512

✓

Vista (Ours)

1740h

10 Hz

576

×

1024

✓

✓

✓

✓

DriveSim

(2016/08)

80

×

160

DriveDreamer

(2023/09)

128

×

192

Drive-WM

(2023/11)

192

×

384

(2023/12)

256

×

448

GenAD

(2024/03)

256

×

448

GAIA-1

(2023/09)

288

×

512

576

×

1024

DriveGAN

(2021/04)

256

×

256

Vista (Ours)

To this end, we introduce Vista , a driving world model that is proficient in cross-domain generalization, high-fidelity prediction, and multi-modal action controllability. Specifically, we develop the predictive model on a large corpus of worldwide driving videos [136] to foster its generalization ability. To enable coherent future extrapolation, we condition Vista on three essential dynamic priors (Sec. 3.1). Instead of solely relying on the standard diffusion loss [5], we introduce two explicit loss functions to enhance dynamics and preserve structural details (Sec. 3.1), promoting Vista's ability to simulate realistic futures at high resolution. For flexible controllability, we incorporate a versatile set of action formats, including both high-level intentions such as commands and goal points, as well as low-level maneuvers like trajectories, steering angles, and speeds. These action conditions are injected via a unified interface, which is learned through an efficient training strategy (Sec. 3.2). Consequently, as Fig. 2 shows, Vista acquires the ability to anticipate realistic futures at 10 Hz and 576 × 1024 pixels, and obtains versatile action controllability across various levels of granularity. We also demonstrate the potential of Vista as a generalizable reward function to evaluate the reliability of different actions.

Our contributions are three-fold: (1) We present Vista , a generalizable driving world model that can predict realistic futures at high spatiotemporal resolution. Its prediction fidelity is greatly improved by two novel losses that capture dynamics and preserve structures, along with exhaustive dynamic priors to sustain consistency in long-horizon rollouts. (2) Propelled by an efficient learning strategy, we integrate versatile action controllability into Vista through a unified conditioning interface. The action controllability of Vista can also generalize to different domains in a zero-shot manner. (3) We conduct comprehensive experiments across multiple datasets to verify the effectiveness of Vista. It outperforms the most competitive general-purpose video generator and sets a new state-of-the-art on nuScenes. Our empirical evidence shows that Vista can be used as a reward function to assess actions.

2 Preliminary

We initialize Vista with the pretrained Stable Video Diffusion (SVD) [5], a latent diffusion model for image-to-video generation. For sampling flexibility, SVD adopts a continuous-timestep formula [66, 111]. It converts data samples x to noise n through a diffusion process p ( n | x ) ∼ N ( x , σ 2 I ) , and generates new samples by progressively denoising the latent towards σ 0 = 0 from Gaussian noise. The training of SVD can be simplified to minimizing E x ,σ, n [ λ σ ∥ D θ ( n ; σ ) -x ∥ 2 ] , where D θ is a parameterized UNet denoiser and λ σ is a re-weighting function omitted hereinafter for brevity. Based on this framework, SVD processes a sequence of noisy latent n = { n 1 , n 2 , ..., n K } ∈ R K × C × H × W and generates a video with K = 25 frames. The generation process is guided by a condition image, whose latent is concatenated channel-wise to the inputs, serving as a reference for content generation.

Despite the high aesthetic quality, SVD lacks several key properties to function as a driving world model. As shown in Sec. 4, the first frame predicted by SVD is not identical to the condition image, making it impractical for autoregressive rollout due to content inconsistency. In addition, SVD struggles with the intricate dynamics of driving scenarios, entailing implausible motions. Moreover,

'go forward'

0s

15s

Reward: 0.872 0.815

Reward: 0.870 0.849

Reward: 0.872 0.832

Reward: 0.888 0.860

(A) High-Fidelity Future Prediction

(B) Coherent Long-Horizon Rollout

(C) Versatile Action Controllability

(D) Generalizable Reward

SVD cannot be controlled by any action format. In contrast, we aim to build a generalizable driving world model that predicts high-fidelity futures with realistic dynamics. It ought to be continuously extendable to long horizons and flexibly controllable by multi-modal actions as illustrated in Fig. 2.

3 Learning a Generalizable Driving World Model

As depicted in Fig. 3, Vista adopts a two-phase training pipeline. First, we build a dedicated predictive model, which involves a latent replacement approach to enable coherent future prediction and two novel losses to enhance fidelity (Sec. 3.1). To ensure the generalization to unseen scenarios, we utilize the largest public driving dataset [136] for training. In the second phase, we incorporate multi-modal actions to learn action controllability with an efficient and collaborative training strategy (Sec. 3.2). Using the ability of Vista, we further introduce a generalizable approach to evaluate actions (Sec. 3.3).

3.1 Phase One: Learning High-Fidelity Future Prediction

Basic Setup. Since world models are initiated to predict futures from the current state, the starting of their prediction should be firmly aligned with the condition image. Therefore, we tailor SVD into a dedicated predictive model by imposing the first frame as the condition image and discarding the noise augmentation [5, 49] during training. With this prediction ability, Vista can perform long-term rollouts by iteratively predicting short-term clips and resetting the condition image with the last clip.

Dynamic Prior Injection. Nevertheless, using the aforementioned setup for training often results in irrational dynamics with respect to historical frames, especially in long-term rollouts. We conjecture that this mainly arises from the ambiguity caused by insufficient priors about the tendency of future motions, which is also a common limitation of existing driving world models [54, 68, 125, 127, 136].

Estimating coherent futures requires at least three essential priors that inherently govern the future motion of instances in the scene: position, velocity, and acceleration. Since velocity and acceleration are the first- and second-order derivative of position respectively, these priors can be entirely derived by using three consecutive frames for conditioning. Concretely, we build a frame-wise mask m ∈ { 0 , 1 } K with a length of K to indicate the presence of condition frames. The mask is set sequentially following the time order, with at most three elements being assigned as 1 to denote three condition

high-level

low-level

command goal point

trajectory

angle

speed

Vista

time

initial frame

dynamic priors

future prediction

multi-modal action controls

zero-init projection

autoregressive rollout

replace

repeat & concat

Vista

Vista

LoRA

video

input

video

output

projection

video

input

video

output

action input

1

st

training phase

2

nd

training phase

+

[STOP

个

frames. Instead of concatenating additional channels to the inputs, we inject new condition frames by replacing the corresponding noisy latent n i with the clean latent z i encoded by the image encoder. Formally, the input latent is constructed as ˆ n = m · z +(1 -m ) · n (see Fig. 3 [Left]). To discern the clean latent, we duplicate a new timestep embedding from the pretrained weights and allocate it to the condition frames according to m . The timestep embeddings for condition frames and prediction frames are trained separately. Compared to channel-wise concatenation, we find that replacing the latent is more effective and flexible in absorbing varying numbers of condition frames. In addition, we observe that the replaced latent, when applied to SVD directly, does not degrade its generation quality. Thus, the original performance will not be disturbed when the training is launched. Since there is no need to predict the observed condition frames, we exclude them from the loss as follows:

where D θ is the UNet denoiser that shares the same architecture with SVD. With the replaced latent holding sufficient priors, Vista can fully capture the status of the surrounding instances and predict more coherent and plausible long-term futures through iterative rollouts. In practice, we leverage the last three frames of a predicted clip as dynamic priors for the next prediction step during rollouts.

Dynamics Enhancement Loss. Unlike general videos that cover rather small spaces, driving videos capture much larger scenes [136]. In most driving videos, distant and monotonous regions dominate the view, with the moving foreground instances only occupying a relatively small area [17]. However, the latter often exhibit higher stochasticity, complicating their prediction. Since Eq. (1) supervises all outputs uniformly, it cannot effectively discriminate the nuances of different regions as Fig. 4(b) shows. As a result, the model cannot efficiently learn to predict realistic dynamics in crucial regions.

As the discrepancy between two adjacent frames provides considerable motion patterns [123, 132], we introduce an additional supervision to encourage the learning of dynamics for crucial regions. To be specific, we first introduce a dynamics-aware weight w = { w 2 , w 3 , ..., w k } ∈ R K -1 × C × H × W that highlights the regions where the prediction has inconsistent motion compared to the ground truth:

For numerical stability, we normalize w within each video clip. As shown in Fig. 4(c), the weight amplifies the presence of large motion disparities, highlighting dynamic regions while excluding monotonous backgrounds. Given the causality of future prediction, i.e . subsequent frames ought to follow previous ones, we define a new loss by penalizing the latter frame of each adjacent frame pair:

where sg ( · ) stops the gradient. By adaptively re-weighting the standard diffusion loss, L dynamics can boost the learning efficiency of dynamic regions, e.g ., the moving vehicles and sidewalks in Fig. 4(d).

Structure Preservation Loss. The trade-off between perceptual quality and motion intensity has been widely acknowledged in video generation [3, 32, 73, 144], and our case is no exception. When

original frames

decoded HF features

(a)

standard diffusion loss

(b)

dynamics-aware weights

(c)

dynamics enhancement loss

(d)

(e)

=

⊙

=

=

⊙

⊙

1

1

一

一

it comes to high-resolution prediction for dynamic driving scenarios, we discover that the predicted structural details degrade severely with over-smoothed or broken objects, e.g ., the outlines of vehicles unravel quickly as they move (see Fig. 12). To alleviate this problem, it is important to place more emphasis on structural details. Based on the fact that structural details, such as edges and textures, mainly reside in high-frequency components, we identify them in the frequency domain as follows:

where FFT and IFFT are the 2D discrete Fourier transform and inverse discrete Fourier transform respectively, and H is an ideal 2D high-pass filter that truncates low-frequency components under a certain threshold. The Fourier transforms are applied on each channel of z i independently. As illustrated in Fig. 4(e), features associated with structural information can be effectively emphasized by Eq. (4). The corresponding features from the predicted latent D θ (ˆ n i ; σ ) can also be extracted similarly. With the extracted high-frequency features, we devise a new structure preservation loss as:

This loss function minimizes the disparity of high-frequency features between prediction and ground truth, so that more structural information can be retained. Our final training objective is a weighted sum of Eq. (1), Eq. (3) and Eq. (5), where λ 1 and λ 2 are trade-off weights to balance the optimization:

3.2 Phase Two: Learning Versatile Action Controllability

Unified Conditioning of Versatile Actions. To maximize usage flexibility, a driving world model should be able to leverage multiple action formats with different characteristics. For instance, one may use the world model to evalute high-level policies [127], or to execute low-level maneuvers [102]. However, existing approaches only support limited action controls [54, 61, 90, 125, 127], inhibiting their flexibility and applicability. Therefore, we incorporate a versatile set of action modes for Vista: (1) Angle and Speed stand for the utmost fine-grained action controls. We normalize angles to [ -1 , 1] and represent speeds in km/h . The signals from different timestamps are concatenated sequentially. (2) Trajectory is a series of 2D displacements in ego coordinates. It is widely used as the output of planning algorithms [12, 21, 58, 62, 63]. We represent the trajectory in meters and flatten it into a sequence. (3) Command is the most high-level intention. Without loss of generality, we define four commands, i.e . go forward, turn right, turn left, and stop, which are implemented as categorical indices. (4) Goal Point is a 2D coordinate projected from the short-term ego destination onto the initial frame, serving as an interactive interface [74]. The coordinate is normalized by the image size.

Note that these actions are heterogeneous and cannot be used interchangeably. After transforming all these actions into numerical sequences, we encode them as a unified concatenation of Fourier embeddings [114, 116] (see Fig. 3). These embeddings can be jointly ingested by learning additional projections to expand the input dimension of the cross-attention layers in the UNet [5]. The new

projections are initialized as zeros to enable gradual learning from the pretrained state. We empirically discover that incorporating action conditions through cross-attention layers yields faster convergence and stronger controllability compared to other approaches such as additive embeddings [128, 136].

Efficient Learning. We learn action controllability after the first training phase. Since the number of total iterations is crucial for diffusion training [5, 22, 32, 99], we separate action control learning into two stages. In the first stage, we train our model at a low resolution (320 × 576), which achieves 3.5 × higher training throughput compared to the original resolution (576 × 1024). This stage constitutes the majority of training iterations. Then, we finetune the model at the desired resolution (576 × 1024) for a short duration, so that the learned controllability can effectively cater to high-resolution prediction.

However, tuning the UNet [5] at a lower resolution directly may undermine the high-fidelity prediction ability. Conversely, freezing all UNet weights and training the new projections alone would precipitate a quality decline (see Appendix D), suggesting the necessity to make the UNet adaptable. To solve this, we freeze the pretrained UNet and introduce parameter-efficient LoRA adapters [55] to each attention layer. After training, the low-rank matrices can be seamlessly integrated with the frozen weights, without introducing extra inference latency. Thus, the pretrained weights remain intact when training at the low resolution, avoiding deterioration of the pretrained high-fidelity prediction ability.

Since the parameters of the camera and vehicle are unavailable for open-world scenarios, it seems impossible to obtain multiple equivalent action conditions simultaneously at inference time. Additionally, it will entail prohibitively expensive training to encompass all possible combinations of action conditions. Hence, unlike common practices that activate all conditions during training, we enforce the independence of different action formats by enabling only one of them for each training sample. The remaining action conditions will be filled with zeros as unconditional inputs. As demonstrated in Appendix D, this simple constraint prevents the squandering of training cost on action combinations and maximizes the learning efficiency of each individual action mode within the same training steps.

Collaborative Training. Note that the aforementioned action conditions are not available in OpenDVYouTube [136]. On the other hand, nuScenes [10] has adequate annotations to derive these conditions. To maintain generalization and learn controllability in tandem, we introduce a collaborative training strategy by utilizing the samples from both datasets, with the action conditions for OpenDV-YouTube set to zero. The action control learning phase adopts the same loss as Eq. (6). By learning from two complementary datasets, Vista gains versatile controllability that are generalizable to novel datasets.

3.3 Generalizable Reward Function

One application of world models is to evaluate actions by engaging a reward module [40, 42, 43, 76]. Drive-WM [127] establishes a reward using external detectors [82, 84]. However, these detectors are developed on a particular dataset [10], which may become a bottleneck for reward estimation in arbitrary scenarios. On the other hand, Vista has ingested millions of human driving logs, exhibiting strong generalization across scenes. Based on the observation that out-of-distribution conditions will lead to increased diversity in generation [28, 60], we utilize the prediction uncertainty from Vista itself as the source of our reward. Different from Drive-WM, our reward function seamlessly inherits the generalization of Vista without resorting to external models. Specifically, we estimate uncertainty via conditional variance. For reliable approximation, we denoise from randomly sampled noise with the same condition frame c and action a for M rounds. Our reward function R ( c , a ) is then defined as the exponential of averaged negative conditional variance:

where avg ( · ) averages all latent values within the video clip. Based on this formulation, unfavorable actions with larger uncertainties will lead to lower rewards. In contrast to commonly used evaluation protocols ( e.g ., the L2 error), our reward function can evaluate actions without referring to the ground truth actions. Note that we do not normalize the estimated rewards for the simplicity of definition, but it is straightforward to amplify the relative contrast by rescaling the estimated rewards with a factor.

Vista

SVD

I2VGen-XL

DynamiCrafter

condition frame

misaligned

misaligned

corrupted

×

448, 2.5s)

ADriver-I (2Hz, 256

Vista (10Hz, 576

×

1024, 15s)

Drive-WM (2Hz, 192

×

384, 8s)

DriveDreamer (12Hz, 128

×

192, 4s), GenAD (2Hz, 256

×

448, 4s)

×

SVD

512, 3.5s)

4 Experiments

In this section, we first demonstrate Vista's strengths in generalization and fidelity in Sec. 4.1. We then show the impact of action controls in Sec. 4.2. We also substantiate the efficacy of the proposed reward function in Sec. 4.3. Finally, we conduct ablation studies on our key designs in Sec. 4.4. For more implementation details and experimental results, please refer to Appendix C and Appendix D.

4.1 Comparisons of Generalization and Fidelity

Automatic Evaluation. Since none of the driving world models are publicly accessible, we compare these methods with their quantitative results on nuScenes. We filter 5369 valid samples from the validation set to conduct FID [47] and FVD [115] evaluation. For FID evaluation, we crop and resize the predicted frames to the resolution of 256 × 448. For FVD evaluation, we use all 25 frames in each video clip and downsample them to 224 × 224 following LVDM [46]. Table 2 reports the results of all methods. In both metrics, Vista surpasses previous driving world models with a considerable margin.

Human Evaluation. To analyze the generalization of Vista across different datasets, we compare it against three prominent general-purpose video generators trained on web-scale data [5, 133, 144] (see Fig. 5). It is known that automatic metrics like FVD [115] cannot conclusively reveal perceptual

96.67

93.33

78.18

93.33

84.85

72.73

3.33

6.67

21.82

6.67

15.15

27.27

I2VGen-XL

DynamiCrafter

SVD

I2VGen-XL

DynamiCrafter

SVD

Visual Quality                             Motion Rationality

Ours

Baselines

Average Subset FVD↓ on nuScenes

action-free

w/ goal point

w/ command

w/ angle & speed

w/ trajectory

action-free

action-free

w/ command

w/ trajectory

Average Subset FVD↓ on Waymo

367.2

322.7

292.2

230.0

207.6

311.8

273.7

268.9

'stop'

angle & speed

trajectory

command

goal point

'turn left'

quality [3, 6, 32, 130, 136], let alone real-world dynamics. Therefore, we opt for human evaluation for more faithful analysis. Following recent advances [3, 5, 6, 15, 16, 32, 122, 126], we adopt the Two-Alternative Forced Choice protocol. Specifically, participants are presented with a side-by-side video pair and asked to choose the video they deem better on two orthogonal aspects: visual quality and motion rationality. To avoid potential bias, we crop each video to a fixed aspect ratio, downsample them to the same resolution, and trim the excess frames when Vista generates longer videos than others. We only feed one condition frame to align with other models. To ensure the variety of scenes, we uniformly assemble 60 scenes from four representative datasets: OpenDV-YouTube-val [136], nuScenes [10], Waymo [112], and CODA [79]. These datasets collectively exemplify the intricacy and diversity of real-world driving, e.g ., OpenDV-YouTube-val includes geofenced districts, Waymo offers a unique domain compared to our training data, and CODA contains extremely challenging corner cases. We collect a total of 2640 answers from 33 participants. As presented in Fig. 7, Vista outperforms all baselines on both aspects, demonstrating its profound comprehension of the driving dynamics. Further, unlike other models that are only applicable for short-term generation, Vista can accommodate more dynamic priors and produce coherent long-horizon rollouts as shown in Fig. 6.

4.2 Results of Action Controllability

Quantitative Results. To evaluate the impact of action controls, we divide the validation set of both nuScenes and the unseen Waymo dataset into four subsets according to our command categories. We then generate predictions using different modalities of the ground truth actions. The FVD score is measured on each subset and then averaged. A lower FVD score reflects a closer distribution to the ground truth videos, indicating that the predictions exhibit more resemblance to each specific type of behavior. Fig. 8 shows that our action controls can emulate the corresponding movements effectively.

We also introduce a new metric named Trajectory Difference to assess control consistency. Following GenAD [136], we train an inverse dynamics model (IDM) that estimates the corresponding trajectory from a video clip. An illustration of IDM is shown in Fig. 13. We then send Vista's prediction to the IDM and calculate the L2 difference between the ground truth trajectory and the estimated trajectory.

Ground Truth

Action1 L2 Error: 0.94  Reward: 0.88

Action2 L2 Error: 1.36  Reward: 0.90

ground

truth

with

1 prior

with

2 priors

with

3 priors

0.880

AverageReward

0.875

0.870

0.865

0.860

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Average L2Error

w/o structure loss

w/ structure loss

w/o dynamics loss

w/ dynamics loss

Effect of Dynamics Enhancement Loss

Effect of Structure Preservation Loss

start frame

zoom-in

The differences are measured over 2 seconds. The lower the trajectory difference, the stronger the control consistency Vista exhibits. We conduct the experiments on nuScenes and Waymo. For each dataset, we collect a subset that contains 537 samples. As reported in Table 3, Vista can be effectively controlled by different modalities of actions, resulting in more consistent motions to the ground truth.

Qualitative Results. Fig. 9 exhibits the versatile action controllability of our model. Vista can be effectively controlled by multi-modal actions, even in unseen scenarios beyond the training domain. In Appendix E, we also showcase the counterfactual reasoning ability of Vista using abnormal actions.

4.3 Results of Reward Modeling

To validate the efficacy of our reward function, we jitter the ground truth trajectories into a series of inferior trajectories. Specifically, we compute the standard deviation of each waypoint from the nuScenes training set as prior distributions. These priors are jointly rescaled to sample perturbations with different L2 errors. The perturbations are then added as offsets to the ground truth trajectories. To ensure the plausibility of sampled trajectories, we adopt an explicit correlating strategy [35, 95] to regularize offset sampling and recursively sample new trajectories until their offsets are consistent in tendencies. To demonstrate the generality of our reward function, we conduct reward estimation on Waymo [112], which is unseen in training. This is done by uniformly sampling from each command category on Waymo validation set, resulting in 1500 cases in total. We compare the average reward of the trajectories with varying L2 errors in Fig. 10. Our reward decreases when the deviation from the ground truth increases, underscoring the potential of our approach to serve as a viable reward function. It also holds the promise to remedy the irrationality in current evaluation protocols for planning [18, 83, 141], such as the L2 error shown in Fig. 10. More in-depth analysis of rewards, including sensitivity to hyperparameters and reward of other actions, are provided in Appendix D.

4.4 Ablation Study

Dynamic Priors. Fig. 11 visualizes the outcomes of using different orders of dynamic priors. The order of priors corresponds to the number of condition frames. It shows that dynamic priors play a pivotal role in long-horizon rollouts, where the coherence with respect to historical frames is essential.

To further demonstrate the efficacy of dynamic priors, we conduct a quantitative evaluation in Table 3. Specifically, we use the IDM in Sec. 4.2 to infer the trajectories of the predicted videos with different orders of dynamic priors. The diminishing differences in trajectory suggest that introducing more priors can effectively improve the consistency between prediction and ground truth.

Auxiliary Supervisions. To verify the effectiveness of the two losses proposed in Sec. 3.1, we devise two additional variants by individually ablating each loss from a variant that incorporates both losses. We qualitatively compare their effects through Fig. 12, which confirms that the dynamics enhancement loss can promote the learning of real-world dynamics, and the structure preservation loss can reinforce the prediction of structural details.

5 Conclusion

In this paper, we introduce Vista , a generalizable driving world model with enhanced fidelity and controllability. Based on our systematic investigations, Vista is able to predict realistic and continuous futures at high spatiotemporal resolution. It also possesses versatile action controllability that is generalizable to unseen scenarios. Moreover, it can be formulated as a reward function to evaluate actions. We hope Vista will usher in broader interest in developing generalizable autonomy systems.

Limitations and future work. As an early endeavor, Vista still exhibits some limitations with respect to computation efficiency, quality maintenance, and training scale. Our future work will look into applying our method to scalable architectures [54, 97]. More discussions are included in Appendix A.

Acknowledgments

This work is supported by National Key R&D Program of China (2022ZD0160104), National Natural Science Foundation of China (62206172), and Shanghai Committee of Science and Technology (23YF1462000). This work is also partially supported by the BMBF (Tübingen AI Center, FKZ: 01IS18039A), the DFG (SFB 1233, TP 17, project number: 276693517), and the EXC (number 2064/1 - project number: 390727645). We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Kashyap Chitta. We also appreciate Zetong Yang, Chonghao Sima, Linyan Huang, and the rest members from OpenDriveLab for valuable feedback. We express our sincere gratitude to all anonymous participants for helping with the human evaluation.

Inverse

Dynamics

Model

Vista

future prediction

𝒙

priors

from history

Trajectory

Difference

trajectory,

command,

goal point,

etc.

multi-modal

action controls

GT Traj.

Pred Traj.

References

Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional Foundation Models for Hierarchical Planning. In NeurIPS , 2023. 19

Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video Pretraining (VPT): Learning to Act by Watching Unlabeled Online Videos. In NeurIPS , 2022. 19

Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: A Space-Time Diffusion Model for Video Generation. arXiv preprint arXiv:2401.12945 , 2024. 4, 8

Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models. In ICLR , 2024. 19

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets. arXiv preprint arXiv:2311.15127 , 2023. 2, 3, 5, 6, 7, 8, 21, 22, 23, 24, 26

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models. In CVPR , 2023. 8, 20, 21

Daniel Bogdoll, Yitian Yang, and J Marius Zöllner. MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations. arXiv preprint arXiv:2311.11762 , 2023. 21

Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to Follow Image Editing Instructions. In CVPR , 2023. 20

Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel. Genie: Generative Interactive Environments. arXiv preprint arXiv:2402.15391 , 2024. 19, 20, 21

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yuxin Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A Multimodal Dataset for Autonomous Driving. In CVPR , 2020. 6, 8, 20, 21, 23, 26

Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuPlan: A Closed-Loop ML-based Planning Benchmark for Autonomous Vehicles. In CVPR Workshops , 2021. 20

Sergio Casas, Abbas Sadat, and Raquel Urtasun. MP3: A Unified Model to Map, Perceive, Predict and Plan. In CVPR , 2021. 1, 5, 21

Jun Cen, Chenfei Wu, Xiao Liu, Shengming Yin, Yixuan Pei, Jinglong Yang, Qifeng Chen, Nan Duan, and Jianguo Zhang. Using Left and Right Brains Together: Towards Vision and Language Planning. arXiv preprint arXiv:2402.10534 , 2024. 19

Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by Cheating. In CoRL , 2019. 1, 21

Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafter1: Open Diffusion Models for High-Quality Video Generation. arXiv preprint arXiv:2310.19512 , 2023. 8, 21

Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models. In CVPR , 2024. 8

Kai Chen, Enze Xie, Zhe Chen, Yibo Wang, Lanqing Hong, Zhenguo Li, and Dit-Yan Yeung. GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation. In ICLR , 2024. 4

Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-End Autonomous Driving: Challenges and Frontiers. arXiv preprint arXiv:2306.16927 , 2023. 1, 9, 19

Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. GenTron: Delving Deep into Diffusion Transformers for Image and Video Generation. In CVPR , 2024. 20

Kashyap Chitta, Daniel Dauner, and Andreas Geiger. SLEDGE: Synthesizing Simulation Environments for Driving Agents with Generative Models. arXiv preprint arXiv:2403.17933 , 2024. 21

Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. TransFuser: Imitation with Transformer-based Sensor Fusion for Autonomous Driving. IEEE TPAMI , 2023. 1, 5, 21

Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, and Devi Parikh. Emu: Enhancing Image Generation Models using Photogenic Needles in a Haystack. arXiv preprint arXiv:2309.15807 , 2023. 6

Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking. arXiv preprint arXiv:2406.15349 , 2024. 20

Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis. In NeurIPS , 2021. 21

Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning Universal Policies via Text-Guided Video Generation. In NeurIPS , 2023. 19

Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, brian ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tompson. Video Language Planning. In ICLR , 2024. 19, 21

Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual Foresight: Model-based Deep Reinforcement Learning for Vision-based Robotic Control. arXiv preprint arXiv:1812.00568 , 2018. 19, 21

Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video Prediction Models as Rewards for Reinforcement Learning. In NeurIPS , 2023. 6, 19

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for High-Resolution Image Synthesis. In CVPR , 2021. 21

Chelsea Finn and Sergey Levine. Deep Visual Foresight for Planning Robot Motion. In ICRA , 2017. 19, 21

Zeyu Gao, Yao Mu, Ruoyan Shen, Chen Chen, Yangang Ren, Jianyu Chen, Shengbo Eben Li, Ping Luo, and Yanfeng Lu. Enhance Sample Efficiency and Robustness of End-to-End Urban Autonomous Driving via Semantic Masked World Model. In NeurIPS Workshops , 2022. 21

Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning. arXiv preprint arXiv:2311.10709 , 2023. 4, 6, 8, 20, 21

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In NeurIPS , 2014. 21

Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning. In ICLR , 2024. 21

Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martín, and Li Fei-Fei. MaskViT: Masked Visual Pre-Training for Video Prediction. In ICLR , 2023. 9, 19, 21, 23

Tarun Gupta, Wenbo Gong, Chao Ma, Nick Pawlowski, Agrin Hilmkil, Meyer Scetbon, Ade Famoti, Ashley Juan Llorens, Jianfeng Gao, Stefan Bauer, Bernhard Kragic, Danica an Schölkopf, and Cheng Zhang. The Essential Role of Causality in Foundation World Models for Embodied AI. arXiv preprint arXiv:2402.06665 , 2024. 25

Wes Gurnee and Max Tegmark. Language Models Represent Space and Time. In ICLR , 2024. 21

Nicholas Guttenberg and CrossLabs. Diffusion with Offset Noise, 2023. 22

David Ha and Jürgen Schmidhuber. Recurrent World Models Facilitate Policy Evolution. In NeurIPS , 2018. 21

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to Control: Learning Behaviors by Latent Imagination. arXiv preprint arXiv:1912.01603 , 2019. 6, 21

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning Latent Dynamics for Planning from Pixels. In ICML , 2019. 21

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with Discrete World Models. In ICLR , 2021. 6, 21

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering Diverse Domains through World Models. arXiv preprint arXiv:2301.04104 , 2023. 6, 21

Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with Language Model is Planning with World Model. In EMNLP , 2023. 21

Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning. arXiv preprint arXiv:2402.14407 , 2024. 21

Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent Video Diffusion Models for High-Fidelity Long Video Generation. arXiv preprint arXiv:2211.13221 , 2022. 7, 21

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NeurIPS , 2017. 7

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In NeurIPS , 2020. 21

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded Diffusion Models for High Fidelity Image Generation. JMLR , 2022. 3, 20, 22

Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. arXiv preprint arXiv:2207.12598 , 2022. 22

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video Diffusion Models. arXiv preprint arXiv:2204.03458 , 2022. 21

Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based Imitation Learning for Urban Driving. In NeurIPS , 2022. 21

Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras. In ICCV , 2021. 19, 21

Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. GAIA-1: A Generative World Model for Autonomous Driving. arXiv preprint arXiv:2309.17080 , 2023. 1, 2, 3, 5, 10, 20, 21

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR , 2022. 6, 22

Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning. In ECCV , 2022. 1, 21, 22

Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Tianyi Zhang, Shibo Zhao, Yu Quan Chong, Chen Wang, Katia Sycara, Matthew Johnson-Roberson, Dhruv Batra, Xiaolong Wang, Sebastian Scherer, Zsolt Kira, Fei Xia, and Yonatan Bisk. Toward GeneralPurpose Robots via Foundation Models: A Survey and Meta-Analysis. arXiv preprint arXiv:2312.08782 , 2023. 1, 20

Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. PlanningOriented Autonomous Driving. In CVPR , 2023. 1, 5, 21, 22

Zhiting Hu and Tianmin Shu. Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning. arXiv preprint arXiv:2312.05230 , 2023. 21

Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffusion Reward: Learning Rewards via Conditional Video Diffusion. arXiv preprint arXiv:2312.14134 , 2023. 6, 19

Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. ADriver-I: A General World Model for Autonomous Driving. arXiv preprint arXiv:2311.13549 , 2023. 1, 2, 5, 21

Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving. In ICCV , 2023. 5

Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think Twice Before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving. In CVPR , 2023. 5

Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. VAD: Vectorized Scene Representation for Efficient Autonomous Driving. In ICCV , 2023. 1, 21, 22

Michael I Jordan and David E Rumelhart. Forward Models: Supervised Learning with a Distal Teacher. Cognitive Science , 1992. 19

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion-based Generative Models. In NeurIPS , 2022. 2, 22

Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting. In CVPR , 2023. 21

Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. DriveGAN: Towards a Controllable High-Quality Neural Simulation. In CVPR , 2021. 2, 3, 21

Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to Simulate Dynamic Environments with GameGAN. In CVPR , 2020. 21

Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv preprint arXiv:1312.6114 , 2013. 21

Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to Act from Actionless Videos through Dense Correspondences. In ICLR , 2024. 19

Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A World Model for Indoor Navigation. In ICCV , 2021. 21

Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, Yong Cheng, Ming-Chang Chiu, Josh Dillon, Irfan Essa, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, David Ross, Grant Schindler, Mikhail Sirotenko, Kihyuk Sohn, Krishna Somandepalli, Huisheng Wang, Jimmy Yan, Ming-Hsuan Yang, Xuan Yang, Bryan Seybold, and Lu Jiang. VideoPoet: A Large Language Model for Zero-Shot Video Generation. arXiv preprint arXiv:2312.14125 , 2023. 4

Hanyang Kong, Dongze Lian, Michael Bi Mi, and Xinchao Wang. DreamDrone. arXiv preprint arXiv:2312.08746 , 2023. 5, 21

Lei Lai, Zhongkai Shangguan, Jimuyang Zhang, and Eshed Ohn-Bar. XVO: Generalized Visual Odometry via Cross-Modal Self-Training. In ICCV , 2023. 19

Yann LeCun. A Path towards Autonomous Machine Intelligence. Open Review , 62, 2022. 1, 6, 19, 20, 21

Hongyang Li, Yang Li, Huijie Wang, Jia Zeng, Huilin Xu, Pinlong Cai, Li Chen, Junchi Yan, Feng Xu, Lu Xiong, Jingdong Wang, Futang Zhu, Chunjing Xu, Tiancai Wang, Fei Xia, Beipeng Mu, Zhihui Peng, Dahua Lin, and Yu Qiao. Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future. arXiv preprint arXiv:2312.03408 , 2023. 21

Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Jia Zeng, Zhiqi Li, Jiazhi Yang, Hanming Deng, Hao Tian, Enze Xie, Jiangwei Xie, Li Chen, Tianyu Li, Yang Li, Yulu Gao, Xiaosong Jia, Si Liu, Jianping Shi, Dahua Lin, and Yu Qiao. Delving Into the Devils of Bird's-Eye-View Perception: A Review, Evaluation and Recipe. IEEE TPAMI , 2023. 21

Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit-Yan Yeung, Xiaodan Liang, Zhenguo Li, and Hang Xu. CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving. In ECCV , 2022. 8, 26

Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. In ICLR , 2023. 21

Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2). arXiv preprint arXiv:2402.16720 , 2024. 21

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers. In ECCV , 2022. 6, 19

Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? In CVPR , 2024. 1, 9, 19, 21

Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction. In ICLR , 2023. 6, 19

Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to Model the World with Language. arXiv preprint arXiv:2308.01399 , 2023. 21

Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World Model on Million-Length Video and Language With RingAttention. arXiv preprint arXiv:2402.08268 , 2024. 21

Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. arXiv preprint arXiv:1711.05101 , 2017. 22

William Lotter, Gabriel Kreiman, and David Cox. Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning. In ICLR , 2017. 21

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. In NeurIPS , 2022. 20

Jiachen Lu, Ze Huang, Jiahui Zhang, Zeyu Yang, and Li Zhang. WoVoGen: World Volume-Aware Diffusion for Controllable Multi-Camera Driving Scene Generation. arXiv preprint arXiv:2312.02934 , 2023. 1, 2, 5, 7, 21

Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference. arXiv preprint arXiv:2310.04378 , 2023. 20

Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured World Models from Human Videos. In RSS , 2023. 21

Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On Distillation of Guided Diffusion Models. In CVPR , 2023. 20

Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are Sample-Efficient World Models. In ICLR , 2023. 21

Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep Dynamics Models for Learning Dexterous Manipulation. In CoRL , 2020. 9, 23

Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-Conditional Video Prediction using Deep Networks in Atari Games. In NeurIPS , 2015. 21

William Peebles and Saining Xie. Scalable Diffusion Models with Transformers. In ICCV , 2023. 10, 20

AJ Piergiovanni, Alan Wu, and Michael S Ryoo. Learning Real-World Robot Policies by Dreaming. In IROS , 2019. 21

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In ICLR , 2024. 6, 20

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR , 2022. 21

Tim Salimans and Jonathan Ho. Progressive Distillation for Fast Sampling of Diffusion Models. In ICLR , 2023. 20

Eder Santana and George Hotz. Learning a Driving Simulator. arXiv preprint arXiv:1608.01230 , 2016. 2, 5, 7, 21

Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation. arXiv preprint arXiv:2403.12015 , 2024. 20

Ingmar Schubert, Jingwei Zhang, Jake Bruce, Sarah Bechtle, Emilio Parisotto, Martin Riedmiller, Jost Tobias Springenberg, Arunkumar Byravan, Leonard Hasenclever, and Nicolas Heess. A Generalist Dynamics Model for Control. arXiv preprint arXiv:2305.10912 , 2023. 21

Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-Efficient Reinforcement Learning with Self-Predictive Representations. In ICLR , 2021. 21

Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine. ViNT: A Foundation Model for Visual Navigation. In CoRL , 2023. 19

Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. DriveLM: Driving with Graph Visual Question Answering. arXiv preprint arXiv:2312.14150 , 2023. 20, 21

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Video: Text-to-Video Generation without Text-Video Data. In ICLR , 2023. 21

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In ICLR , 2021. 22

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency Models. In ICML , 2023. 20

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based Generative Modeling through Stochastic Differential Equations. In ICLR , 2021. 2

Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott M. Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. In CVPR , 2020. 8, 9, 20, 25, 26

Richard S Sutton. The Quest for a Common Model of the Intelligent Decision Maker. arXiv preprint arXiv:2202.13252 , 2022. 20, 21

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. In NeurIPS , 2020. 5, 21

Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards Accurate Generative Models of Video: A New Metric & Challenges. arXiv preprint arXiv:1812.01717 , 2018. 7

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In NeurIPS , 2017. 5, 21

Vikram Voleti, Alexia Jolicoeur-Martineau, and Chris Pal. MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation. In NeurIPS , 2022. 21

Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel Multi-View Synthesis and 3D Generation from a Single Image using Latent Video Diffusion. arXiv preprint arXiv:2403.12008 , 2024. 22

Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating Videos with Scene Dynamics. In NeurIPS , 2016. 21

Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation. In ICCV , 2023. 21

Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. ModelScope Text-to-Video Technical Report. arXiv preprint arXiv:2308.06571 , 2023. 21

Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. VideoFactory: Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation. arXiv preprint arXiv:2305.10874 , 2023. 8

Xiang Wang, Shiwei Zhang, Hangjie Yuan, Zhiwu Qing, Biao Gong, Yingya Zhang, Yujun Shen, Changxin Gao, and Nong Sang. A Recipe for Scaling up Text-to-Video Generation with Text-Free Videos. In CVPR , 2024. 4

Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. VideoLCM: Video Latent Consistency Model. arXiv preprint arXiv:2312.09109 , 2023. 20

Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. DriveDreamer: Towards RealWorld-Driven World Models for Autonomous Driving. arXiv preprint arXiv:2309.09777 , 2023. 1, 2, 3, 5, 7, 21

Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, Yuwei Guo, Tianxing Wu, Chenyang Si, Yuming Jiang, Cunjian Chen, Chen Change Loy, Bo Dai, Dahua Lin, Yu Qiao, and Ziwei Liu. LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models. arXiv preprint arXiv:2309.15103 , 2023. 8, 20

Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving. In CVPR , 2024. 1, 2, 3, 5, 6, 7, 19, 20, 21

Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. MotionCtrl: A Unified and Flexible Motion Controller for Video Generation. arXiv preprint arXiv:2312.03641 , 2023. 6, 20, 21

Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting. In NeurIPS Datasets and Benchmarks , 2023. 20

Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge, Xiaodong Cun, David Junhao Zhang, Jia-Wei Liu, Yuchao Gu, Rui Zhao, Weisi Lin, Wynne Hsu, Ying Shan, and Mike Zheng Shou. Towards A Better Metric for Text-to-Video Generation. arXiv preprint arXiv:2401.07781 , 2024. 8

Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-Training Contextualized World Models with In-the-Wild Videos for Reinforcement Learning. In NeurIPS , 2023. 21

Penghao Wu, Li Chen, Hongyang Li, Xiaosong Jia, Junchi Yan, and Yu Qiao. Policy Pre-training for Autonomous Driving via Self-supervised Geometric Modeling. In ICLR , 2023. 4

Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. DynamiCrafter: Animating Open-Domain Images with Video Diffusion Priors. arXiv preprint arXiv:2310.12190 , 2023. 7, 22

Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video Generation using VQ-VAE and Transformers. arXiv preprint arXiv:2104.10157 , 2021. 21

Xu Yan, Haiming Zhang, Yingjie Cai, Jingming Guo, Weichao Qiu, Bin Gao, Kaiqiang Zhou, Yue Zhao, Huan Jin, Jiantao Gao, Zhen Li, Lihui Jiang, Wei Zhang, Hongbo Zhang, Dengxin Dai, and Bingbing Liu. Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities. arXiv preprint arXiv:2401.08045 , 2024. 1

Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, Jun Zhang, Andreas Geiger, Yu Qiao, and Hongyang Li. Generalized Predictive Model for Autonomous Driving. In CVPR , 2024. 2, 3, 4, 6, 7, 8, 20, 21, 23, 25, 26

Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning Interactive Real-World Simulators. In ICLR , 2024. 19, 21

Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the New Language for Real-World Decision Making. arXiv preprint arXiv:2402.17139 , 2024. 20

Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion. arXiv preprint arXiv:2402.03162 , 2024. 21

Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li. Visual Point Cloud Forecasting Enables Scalable Autonomous Driving. In CVPR , 2024. 21

Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes. arXiv preprint arXiv:2305.10430 , 2023. 9, 19

Alex Zhang, Khanh Nguyen, Jens Tuyls, Albert Lin, and Karthik Narasimhan. Language-Guided World Models: A Model-based Approach to AI Control. arXiv preprint arXiv:2402.01695 , 2024. 21

Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion. In ICLR , 2024. 1, 21

Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models. arXiv preprint arXiv:2311.04145 , 2023. 4, 7, 20, 21, 22

Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation. arXiv preprint arXiv:2403.06845 , 2024. 20, 21

Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models. In NeurIPS , 2023. 20

Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving. arXiv preprint arXiv:2311.16038 , 2023. 1, 21

Wenzhao Zheng, Ruiqi Song, Xianda Guo, and Long Chen. GenAD: Generative End-to-End Autonomous Driving. arXiv preprint arXiv:2402.11502 , 2024. 19

Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied Understanding of Driving Scenarios. arXiv preprint arXiv:2403.04593 , 2024. 20

Appendix

F

Licence of Assets

26

A Discussions

To help a thorough understanding of this work, we discuss intuitive questions that might be raised.

Q1. Why is at least position, velocity, and acceleration required to predict coherent futures?

Position ensures the predicted future begins continuously with the current state. Velocity manifests how objects are moving, e.g ., whether they are turning left or turning right. Acceleration represents how velocity changes over time, e.g ., whether the surroundings are moving faster or moving slower. Without utilizing acceleration as a cue, a car overtaking the ego-vehicle may suddenly be passed by in the next autoregressive prediction step. These three priors provide essential cues to allow consistent future extension with respect to historical observation.

Q2. How is the form of the proposed reward function defined?

Unlike VIPER [28] and Diffusion Reward [60] that both make discrete predictions, our model predicts continuous latent. Therefore, our reward is estimated according to conditional variance rather than log-likelihood or entropy. In addition, measuring uncertainty with log-likelihood requires comparing the prediction to the ground truth. As we deploy the reward in any scenario, the approach of VIPER is infeasible for our objective. Note that our reward calculation is meticulously designed to satisfy the Kolmogorov axioms, i.e . it is non-negative and the measure of the entire sample space is [0 , 1] .

Q3. Reward estimation efficiency compared to the detector-based method [127] .

Though our reward estimation involves multi-round denoising, it will not spend more compute than the detector-based reward function defined in Drive-WM [127]. To be specific, Drive-WM obtains the rewards from the perception results. Given that the detectors [82, 84] take image sequences as inputs, Drive-WM has to accomplish all steps of the denoising process before perception. Differently, our reward function estimates the reward with the uncertainty that originates from the world model itself without relying on other perception models. Therefore, the estimation of uncertainty does not require completing the generation process. It can be realized by only denoising each sample for a few steps. In fact, as specified in Appendix C, the total computation required for reward estimation per situation (10 steps, 5 rounds) is no greater than that of generating the entire video (50 steps for our model) as Drive-WM does. Note that the computational cost for our reward estimation can be flexibly reduced to further improve its efficiency. As shown in Fig. 14, using 5 denoising steps ( 50% of the default computation) also yields satisfactory estimations of the reward.

Q4. Usage of the proposed reward function.

(1) As mentioned in Sec. 4.3, the proposed reward function can potentially serve as an alternative metric of driving actions that mitigate the concerns in existing open-loop evaluation [18, 83, 141]. (2) As demonstrated in Fig. 10, better actions generally yield higher rewards with our reward function. Taking advantage of this property, there is great promise for our reward function to be used as a critic module [76], which enables model-predictive control by executing the optimal action that maximizes the estimated reward [27, 30, 35]. This procedure can be performed in conjunction with distribution-based planners [53, 148] that can make action proposals to reduce the searching space.

Q5. Any other potential applications of Vista?

(1) As a generalizable predictive model, Vista could be utilized as a forward dynamics model [13, 26] to simulate short-term dynamics and assist long-horizon planning tasks like visual navigation [106]. (2) It is also intriguing to utilize Vista as an implicit driving policy, which is spontaneously acquired through future prediction [1, 25]. After synthesizing the video plan, we can convert the resultant image trajectory to executable actions by a non-causal inverse dynamics model [4, 65, 71], which can be efficiently learned from much fewer data compared with the imitation learning pipeline [2, 9]. In autonomous driving, the inverse dynamics model could be implemented with visual odometry [75]. (3) In collaboration with the reward function, it is also worth investigating if Vista could facilitate model-based reinforcement learning by boosting the sampling efficiency in real-world scenarios [137].

Q6. Differences with GenAD [136] .

These two works have fundamental differences in control versatility and prediction fidelity. First of all, Vista is a generalizable world model that can be controlled by multi-modal action conditions. Although GenAD has also trained a trajectory-conditioned extension, its weights are fully finetuned on nuScenes and the generalization of its action control has never been validated. In contrast, Vista integrates versatile action controllability that can generalize to new scenarios in a zero-shot manner. Unlike GenAD that requires labeling OpenDV-YouTube with commands and texts, our collaborative training strategy skillfully averts this labor that may incur accumulated noises and conflicts [136]. In addition, Vista (10 Hz, 576 × 1024) operates at much higher frame rate and resolution, considerably beyond the capability of GenAD (2 Hz, 256 × 448) in both temporal and spatial axes. Different from GenAD, we also put forth several dedicated designs for high-fidelity prediction. We find that Vista, with a lower model complexity, achieves much better FID and FVD scores than GenAD (see Table 2).

Q7. Limitations, failure cases, and possible solutions.

As one of the pioneering efforts, Vista still has a few limitations that call for future works. (1) Since Vista predicts futures at an exceptional spatiotemporal resolution, it is inevitable to be computationally expensive, particularly in downstream applications. Potential solutions may include faster sampling techniques [89, 146] and training-based distillations [91, 93, 101, 103, 110, 124]. (2) It is possible that the prediction may undergo an apparent degradation in long-horizon rollouts or drastic view shifts. Extra refinements on the prediction results [6, 49, 54, 99, 126, 144] could be helpful. Speculatively, applying our recipe to more scalable architecture [54, 97] is also promising to address this limitation. (3) Similar to other controllable video generation methods [128], the chance of failure still persists in our action controls, especially for ambiguous intentions such as commands and goal points as Fig. 8 reveals. Incorporating more datasets with action annotations [11, 112, 129] for collaborative training could be beneficial. Using compositional classifier-free guidance [8, 19, 32] to amplify the individual impact of action conditions may also help (at a cost of increased inference compute). (4) Although our training data is based on the largest public driving dataset [136], it is nowhere near the entirety of Internet driving data, thus leaving huge untapped potential to further expand the capabilities of Vista.

Q8. Why not expand the Vista framework to surround-view generation?

It is true that supporting surround-view generation would further help driving. Existing works [127, 145] have explored the surround-view settings on nuScenes [10]. However, in this paper, we focus on the front-view setting for three main reasons: (1) The front view setting allows leveraging diverse data sources [54, 136]. Conversely, the distinctions in multi-view videos from various datasets, such as different numbers of cameras, hinder unified modeling and data scaling. (2) Models that focus on the front view can be seamlessly applied to different datasets without adaptation [107], broadening their applicability across datasets. (3) Though incomplete, the front view often contains most of the information vital for driving. As demonstrated in NA VSIM [23], using the front view alone results in only a 1 . 1% performance downtick in collision rate compared to using five surround-view cameras.

Q9. Broader impact.

Despite the encouraging improvements, our work is by no means perfect when it comes to real-world applications that involve dealing with highly complicated situations. As Vista is based on the diffusion framework, which introduces stochastic outcomes and non-negligible latencies, deploying it into autonomous vehicles directly could pose safety risks. While it is not a silver bullet yet, we expect that Vista will inspire the community to further exploit the capabilities and applications of driving world models. As a prototype for generalizable driving world models, we hope that Vista can stimulate the investigations in developing generalizable systems for autonomous driving and machine intelligence.

B Related Work

B.1 World Models

Intelligent agents should be able to make efficacious decisions even under unseen situations [9, 57, 76, 113, 138, 149]. This requires fundamental knowledge of the world that generalizes to rare cases. As an

internal manifestation of such knowledge, a world model predicts plausible futures of the world given potential actions [9, 40, 69, 76, 96, 113, 137]. In principle, it not only predicts how the environment will unfold over time, but also deduces the underlying physical dynamics and agentic behaviors. Such properties can be useful for representation learning [35, 45, 88, 105, 131], model-based reinforcement learning [39, 40, 42, 43, 94, 96, 98], as well as model-predictive control [27, 30, 35, 41, 92, 104, 142]. Recent methods [37, 44, 80, 85, 86] also induce language-based world models from large language models, but are restricted in textual space and struggle with grounding on physics [26, 59].

Although world models have been extensively applied and made significant revolutions in simulated games [40, 42, 43] and indoor embodiment [72, 92, 120], such investigations for autonomous driving still lag behind [127, 143]. Different from other tasks, world modeling for autonomous driving poses unique challenges, which primarily arise from the large field of views with highly dynamic motions. Some practices imagine the world in the bird's eye view (BEV) space [20, 31, 52, 53, 78, 81]. Recent practices model the world state as raw sensor observations such as point clouds [7, 67, 140, 143, 147] and images [54, 61, 68, 90, 102, 125, 127, 131, 145]. The latter category, namely visual world models, hold more promise for scaling up due to sensor flexibility and data accessibility. Nevertheless, existing methods are restricted to a particular dataset [61, 77, 90, 125, 127, 143, 145, 147] or simulator [7, 131], compromising their generalization ability to novel domains. Meanwhile, these efforts lack systematic designs for the driving domain and only model the world at relatively low frame rates and resolutions, which discards the fine-grained details and impairs their ability to express real-world behaviors. Moreover, most of them are restricted to a specific control modality [54, 61, 90, 125], which hinders the accommodation to prevailing planning algorithms [12, 14, 21, 56, 58, 64] and extension to more applications like decision-making [127] or user interaction [74]. Besides, existing methods seldom explore zero-shot action controllability across different datasets. The inferior generalization, fidelity and controllability collectively preclude existing driving world models from broadly facilitating the development of autonomous driving.

B.2 Video Generation

Video generation is an effective way to model the world and has undergone remarkable advancements over the years. Pioneering works [119, 134] have studied various kinds of generative models [29, 33, 70]. Swayed by the success of diffusion models [24, 48, 100], a surge of diffusion-based video generation methods have emerged [6, 34, 46, 51, 108, 117, 121]. Recent works [5, 15, 32, 144] shift their focus towards image-to-video generation for its finer content description and better scalability in training data. However, most of them are not strict predictive models that generate videos starting from the condition image. Moreover, existing methods struggle with the intricate dynamics in driving scenarios from the ego perspective [136], which limits their feasibility as driving world models.

While the majority of existing methods produce videos without explicit controllability, two recent works [128, 139] introduce camera motion control to video generation. However, camera motion is conceptually distinct from vehicle actions and both of these works are text-to-video methods without any prediction ability. Contrarily, the model we developed is a predictive world model that produces realistic dynamics and allows versatile action controls for autonomous driving.

C Implementation Details

C.1 Model

We adopt the framework of SVD [5] as the architecture of Vista, which consists of 2.5B parameters in total, including 1.6B UNet parameters. For action conditioning, we encode the value of each action sequence into Fourier embeddings [114, 116] with 128 channels.

C.2 Dataset

We utilize a rigorously filtered set of OpenDV-YouTube [136] for training, and incorporate nuScenes training set [10] during the action control learning phase. Concretely, we manually eliminate 15 hours of irrelevant content from OpenDV-YouTube, yielding approximately 1735 hours of unlabeled driving videos. Since nuScenes is heavily biased [83, 107, 127], we balance its samples based on command categories to foster the learning of rare actions. The video clips are sampled with 25 frames at 10 Hz. Although nuScenes [10] is logged at 12 Hz, we find no negative impact of treating them

as 10 Hz videos. The model inputs are composed by cropping and resizing these clips to the target resolution. We define an action as a sequence comprising 25 frames. To categorize actions into commands, we follow the established conventions in planning [56, 58, 64] and define the command of ego-vehicle as " turn right " or " turn left " when its final displacement exceeds 2 meters in the orthogonal direction relative to its initial heading. To allow more precise categorization, we additionally introduce a " stop " command when the forward driving distance is less than 2 meters.

C.3 Training

At the first training phase, we train all UNet parameters at 576 × 1024 resolution on 128 A100 GPUs for 20K iterations, which takes about 8 days in total. We accumulate the gradients of 2 steps, yielding an effective batch size of 256. Following SVD, our model is trained with the EDM framework [66]. We use the AdamW optimizer [87] with a learning rate of 1 × 10 -5 . The learning rate for spatial layers is moderated by a discount factor of 0 . 1 . The coefficients λ 1 and λ 2 in Eq. (6) are set to 1 . 0 and 0 . 1 respectively. Offset noise [38] is also used with a strength of 0 . 02 as it helps improve temporal smoothness. We randomly sample different orders of dynamic priors with increasing probabilities, i.e . 1 / 15 , 2 / 15 , 4 / 15 , 8 / 15 for 0, 1, 2, 3 condition frames respectively. The noise augmentation [49] is disabled to retain more details from the condition frames.

As for the action control learning phase, we freeze the pretrained weights and add LoRA [55] and projection layers to all attention blocks of the UNet. The rank of LoRA is set to 16. We then train the new weights at 320 × 576 resolution for 120K iterations using batch size 8 and learning rate 5 × 10 -5 . After the controllability can be clearly witnessed, we continue to finetune the unfrozen weights at 576 × 1024 resolution for another 10K iterations. We drop out each activated action mode with a ratio of 15% to allow classifier-free guidance [50]. The sampling ratio of OpenDV-YouTube and nuScenes is 1 : 1 at this training phase. The whole training process for action controllability takes around 10 days on 8 A100 GPUs, with roughly 8 days at the low resolution and 2 days at the high resolution.

C.4 Sampling

We generate future videos using the DDIM sampler [109] for 50 steps. The sampling starts with σ max at 700 . 0 . Since our model predicts long-term futures in an autoregressive manner, the issue of over-saturation caused by the standard classifier-free guidance will accumulate rapidly. Therefore, unlike SVD that linearly increases the guidance scale, we employ a triangular classifier-free guidance scheme [118] to permit genuine long-horizon rollouts. Specifically, for the i -th frame in each K frames to predict, we assign its guidance scale s ( i ) following:

where s min and s max indicate the minimum and maximum guidance scales along the temporal axis. In our experiments, we define s min as 1 . 0 and s max as 2 . 5 . This triangle scheme assigns moderate guidance scales to the frames that will be used as conditions in the next prediction round. Due to sufficient temporal interaction, the quality of intermediate frames can also propagate to the frames that have lower guidance scales. As illustrated in Fig. 15, this technique adeptly mitigates the saturation drift problem while enhancing details. To improve perceptual continuity, we split the generated latent into clips with an overlap of 3 frames before sending them to the video-aware decoder [5]. After decoding, the overlapped frames are averaged pixel-wise.

C.5 Human Evaluation

Recall that we ask the participants to judge side-by-side video pairs from visual quality and motion rationality. To guarantee credible responses, we provide detailed commentary for each aspect of the human evaluation. For visual quality, we let the participants focus on the consistency and harmony of the generated content. For motion rationality, we encourage the participants to pay more attention to the plausibility of the ways that ego-vehicle and other agents move, e.g ., whether they are following the traffic rules and exhibiting safe behaviors. For all public models we compared, we use the official checkpoints and configurations for inference without finetuning. For the models that require textual inputs [133, 144], we set the prompt as " realistic drive view ".

C.6 Reward Estimation

For each condition frame and action pair, we accumulate an ensemble with size M = 5 to obtain a reliable uncertainty estimation. Each sample in the ensemble is inferred for 10 denoising steps as we find it is unnecessary to generate high-quality results for uncertainty estimation. The coefficient β in the correlating strategy [35, 95] is set to 0.5.

C.7 Ablation Studies

For the ablation of loss functions, we train each variant on OpenDV-YouTube [136] for 10K steps at a spatial resolution of 576 × 1024. All ablations, including the additional ablations in Appendix D, are initialized by loading the pretrained checkpoint of SVD [5] and conducted with 8 A100 GPUs.

D Additional Experiments

D.1 Parameter Sensitivity of Reward Estimation

To investigate how the number of denoising steps and the ensemble size influence the performance of the proposed reward function, we repeat the reward estimation procedure in Sec. 4.3 with different hyperparameter settings. We start off by using 5 denoising steps and an ensemble size of 5 for each sample. We then test two variants that double the computational cost by increasing the denoising steps to 10 (our default setup in Appendix C) and increasing the ensemble size to 10 respectively. Following Sec. 4.3, we plot the correlation of the estimated rewards with L2 errors for the three variants in Fig. 14. The results show that increasing the number of denoising steps can greatly enlarge the relative contrast of rewards, suggesting that denoising step is a more important factor than ensemble size under the same computation budget for reward estimation.

D.2 Reward Estimation for Commands

To demonstrate that the proposed reward function is also applicable for other actions, we estimate the rewards of ground truth commands from Waymo and compare them with the rewards of random commands. The results in Table 4 suggest that our reward is also competent for command selection.

D.3 Action Independence Constraint

To prove the efficacy of our learning strategy for action control, we conduct a comparison by removing the action independence constraint proposed in Sec. 3.2. We train two variants on nuScenes [10] at the resolution of 320 × 576 pixels for 62K steps. The comparison results are presented in Table 5.

none

constant

linear (SVD)

triangular (Ours)

time

scale

start frame

turn left

turn right

w/ LoRA

D.4 Triangular Guidance Scheme

We further compare the introduced classifier-free guidance scheme with the vanilla scheme and the linear scheme [5] to verify its necessity. Fig. 15 shows that our triangular scaling attains the best trade-off between visual quality and saturation preservation.

D.5 LoRA Adaptation

To show the necessity of applying LoRA in Sec. 3.2, we train two variants at a resolution of 320 × 576 pixels for 30K iterations. With the pretrained UNet weights fixed, we let one variant train LoRA and action projection layers in the attention blocks, while the other adjusts new projection layers only. As shown in Fig. 16, adding LoRA is essential for action control learning.

D.6 Action Control Consistency

In Table 6, we report the complete FVD scores of Fig. 8, which further validates the effectiveness of all kinds of action controls. Note that since our " stop " subset consists of samples where the final displacements are within 2 meters, the goal points typically do not appear in these videos. Hence, for the experiment that uses goal point as action condition on the " stop " subset, most samples are generated in the same way as the action-free mode.

D.7 Human Evaluation with GenAD

To our best knowledge, there is not a driving-specific world model publicly available so far, making it hard to conduct qualitative human evaluation. Hence, we mainly compare Vista against the existing methods with the officially reported FID and FVD scores in Table 2.

To demonstrate the considerable improvements in visual quality and motion rationality, we conduct an extra human evaluation with the state-of-the-art GenAD model [136]. Since GenAD processes a 4-second video each time, we perform autoregressive prediction to extend Vista 's output to 5 seconds and trim the last second to align with GenAD's duration. To avoid any bias caused by resolution and frequency, we downsample the outputs of Vista (576 × 1024 resolution at 10 Hz) to 256 × 448 resolution at 2 Hz. The evaluation follows the same procedure specified in Sec. 4.1.

Wecollect 25 diverse samples from the unseen OpenDV-YouTubeval set and invite 20 volunteers for evaluation. We ask the volunteers to choose the video they deem better. As a result, Vista is preferred in 94 . 4% and 94 . 8% of the time on visual quality and motion rationality respectively. This attests that Vista, in spite of experiencing a large perceptual reduction due to downsam-

(256

×

448,2Hz)

GenAD

(576

×

1024,10Hz)

Vista

- 2s -

- 4s -

starting

frame

pling, still exhibits a significant advantage over GenAD in generation quality. We also compare the predictions of GenAD and Vista in Fig. 17, showing the superiority of Vista in resolution and fidelity.

E Additional Visualizations

E.1 Generalization Ability

We further demonstrate the strong generalization ability of Vista by deploying it to different scenarios in the wild. The results in Fig. 18 and Fig. 19 illustrate that Vista can make high-fidelity predictions in a very diverse range of scenarios.

E.2 Long-Horizon Prediction

In addition to Fig. 6, we provide more qualitative visualizations of long-horizon prediction in Fig. 20. Vista can continuously predict long-term futures with consistent content and motion.

E.3 Action Controllability

We provide more prediction results with different action inputs in Fig. 21. The results on OpenDVYouTube-val [136] and Waymo [112] show that the versatile controllability of Vista can be readily transferred to different domains in a zero-shot manner.

E.4 Counterfactual Reasoning Ability

Counterfactual reasoning ability is one of the emergent abilities of world models [36]. As shown in Fig. 22, Vista can effectively predict the counterfactual consequences caused by abnormal actions.

E.5 Human Evaluation Cases

To demonstrate the diversity of the scenes selected for human evaluation (Sec. 4), we show all cases gathered from OpenDV-YouTube-val [136], nuScenes [10], Waymo [112], and CODA [79] in Fig. 23.

F Licence of Assets

Our training and evaluation utilize the data from four publicly licensed datasets [10, 79, 112, 136]. Our implementation is based on the codebase of SVD [5], which uses the MIT license. The pretrained checkpoint of SVD is distributed under the stable video diffusion non-commercial community license.

2018

0s

15s

'stop'

angle & speed

trajectory

command

goal point

'turn left'

angle & speed

trajectory

command

goal point

'stop'

angle & speed

trajectory

command

goal point

'turn right'

'turn right'

'turn right'

自

ground

truth

abnormal

action

ground

truth

abnormal

action

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately summarize the contributions and reflect our results in Sec. 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See the conclusion and discussions in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See the implementation details in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code for Vista has been attached to the supplementary material. It will be released to public later.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See the experimental details in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our experiments are conducted at an extensive data scale as specified in Sec. 4. We also provide detailed studies about hyperparameter sensitivity in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See the implementation details in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed and understood the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See our discussions in Appendix A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code will be released with well-organized documentation and instructions. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: See our settings in Sec. 4.1 and Appendix C.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper has no potential risks incurred by study participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper presents Vista, a driving world model that predict future driving video based on video diffusion model. In particular, the author introduces the idea of conditioning on prior frames and two domain-specific losses to capture dynamics and preserve structures for driving scenarios. By using LoRA, Vista also supports controllable (via e.g., trajectory, angle/speed) generation in a zero-shot manner. The author evaluates Vista by showing its high-resoluation videos, low FID on nuScenes and high human evaluation scores.

STRENGTHS:
-neat domain-specific improvement on controllable driving video generation.

-the paper is clear and easy to follow.

WEAKNESSES:
-while the author discussed the differences between Vista and GenAD in Appendix A-Q6 (e.g., different controllability design, and results with higher resolution and lower FID), the improvements seem to be on the incremental side. Besides, GenAD additionally supports text command and its qualitative results of GenAD looks as good as Vista.

-the proposed conditioning on the prior frame and the two domain-specific losses and the use of LoRA while being helpful and neat, are not super novel. 

-human evaluation was not conducted between Vista and driving specific generation models (e.g., GenAD).

QUESTIONS:
-Can you provide more discussions on why the proposed methods are very novel when there are already similar work (e.g., GenAD) on this similar task?

-Potential human evaluation results between Vista and GenAD (and/or other driving domain specific generation models)?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nVista is a generalizable driving world model that excels in high fidelity and versatile controllability. By introducing novel losses to enhance the learning of dynamics and structural information, and integrating a unified conditioning interface for diverse action controls, Vista achieves high-resolution predictions and adapts seamlessly to various scenarios in a zero-shot manner. Extensive experiments demonstrate that Vista outperforms state-of-the-art video generators and driving models, showcasing significant improvements in prediction fidelity and action evaluation capabilities​.\n\nSTRENGTHS:\n1. High Fidelity Predictions: Vista achieves accurate and realistic future predictions at high resolutions.\n2. Versatile Action Controllability: Supports diverse control inputs from high-level commands to low-level maneuvers.\n3. Strong Generalization: Seamlessly adapts to diverse and unseen environments in a zero-shot manner.\n4. Superior Performance: Outperforms state-of-the-art models in prediction fidelity and evaluation metrics.\n\nWEAKNESSES:\n1. I have some questions regarding the production of Figure 5. Is the SVD in Figure 5 used as is, or has it been retrained? Are there any other action control modules? Additionally, while the long-term generation of Vista seems consistent, most of the details are quite blurry. I am also confused about the point that \"the prediction of SVD does not commence from the condition image.\"\n\n2. Regarding the misalignment in DynamiCrafter in Figure 4, it is actually because during training of DynamiCrafter , the condition frame is not always the first frame, but is randomly extracted from the video. Therefore, you can see that the fourth column of DynamiCrafter is consistent with the input frame. This is not a misalignment.\n\nI will raise my rating if all of my concerns are well addressed.\n\nQUESTIONS:\nMentioned in the weakness section\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method named Vista, a novel driving world model that addresses limitations in generalization, prediction fidelity, and action controllability. Key contributions include introducing novel loss functions for high-resolution prediction, a latent replacement approach for coherent long-term rollouts, and versatile action controls ranging from high-level commands to low-level maneuvers. Vista demonstrates strong generalization in zero-shot settings and establishes a generalizable reward function for evaluating driving actions. Extensive experiments on multiple datasets demonstrate that Vista outperforms advanced general-purpose video generators in over 70% of comparisons, surpasses the best-performing driving world model by 55% in FID and 27% in FVD, and establishes a generalizable reward function for real-world driving action evaluation.\n\nSTRENGTHS:\n1. The paper introduces a novel approach to driving world models by incorporating high-fidelity predictions and versatile action controllability. This combination addresses existing gaps in generalization, prediction fidelity, and action flexibility, representing a significant step forward in autonomous driving research. \n2. The paper presents a well-designed methodology that systematically addresses the limitations of existing driving world models. The integration of dynamic prior injection and versatile control mechanisms is methodologically sound and effectively implemented.\n3. The paper is well-structured, with clear and logical sections that guide the reader through the problem formulation, methodology, experiments, and conclusions. Figures and tables are used effectively to illustrate key points and results.\n\nWEAKNESSES:\n1. Vista seems cannot to generate surround-view video, this limitation may restrict the method's effectiveness and generalizability in real-world scenarios where a comprehensive 360-degree view is crucial.\n2. The closed-loop evaluation is demonstrated through only a few cases, raising concerns about the robustness and reliability of integrating the closed-loop process with the generative model. Expanding the evaluation to include a broader range of scenarios and detailed performance metrics would help assess the seamless integration of the closed-loop driving process with the generative model.\n3. The paper introduces dynamic prior injection for coherent long-term rollouts, but the implementation details and impact of this component are not sufficiently elaborated. Providing more detailed information on this mechanism, including its implementation, theoretical basis, and specific impact on long-term prediction consistency, would enhance understanding.\n4. The paper lacks a detailed analysis of the computational resources required for training and inference using the Vista framework. The potentially high computational demands could limit the scalability and real-time applicability of the approach.\n\nQUESTIONS:\n1. Can the authors provide more details on the implementation of the dynamic prior injection mechanism? How does it theoretically support coherent long-term rollouts?\n2. Provide additional experimental results demonstrating the model's performance in zero-shot settings across diverse and unseen environments. This would help validate the framework’s generalization capabilities and robustness in real-world applications.\n3. Expanding the Vista framework to support surround-view video generation could significantly enhance its applicability and robustness in real-world autonomous driving scenarios, as a comprehensive 360-degree view is crucial for effective navigation and situational awareness. Additionally, extending the closed-loop evaluation to include a broader range of scenarios and detailed performance metrics would provide a more thorough assessment of the model's robustness and reliability. Such enhancements would further solidify the contributions of this work and pave the way for future advancements. Nonetheless, the current contributions are commendable, and I look forward to seeing future developments in this area.\n\nLIMITATIONS:\nplease refer to weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}