OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving

## Abstract

Abstract Understanding the evolution of 3D scenes is important for effective autonomous driving.
While conventional methods model scene development with the motion of individual instances, world models emerge as a generative framework to describe the general scene dynamics.
However, most existing methods adopt an autoregressive framework to perform next-token prediction, which suffer from inefficiency in modeling long-term temporal evolutions.
To address this, we propose a diffusion-based 4D occupancy generation model, OccSora, to simulate the development of the 3D world for autonomous driving.
We employ a 4D scene tokenizer to obtain compact discrete spatial-temporal representations for 4D occupancy input and achieve high-quality reconstruction for long-sequence occupancy videos.
We then learn a diffusion transformer on the spatial-temporal representations and generate 4D occupancy conditioned on a trajectory prompt.
We conduct extensive experiments on the widely used nuScenes dataset with Occ3D occupancy annotations.
OccSora can generate 16s-videos with authentic 3D layout and temporal consistency, demonstrating its ability to understand the spatial and temporal distributions of driving scenes.
With trajectory-aware 4D generation, OccSora has the potential to serve as a world simulator for the decision-making of autonomous driving.
Code is available at: https://github.com/wzzheng/OccSora.

## 1 Introduction

As a promising application of artificial intelligence technology, autonomous driving has garnered widespread attention and research in recent years ; ; .
Establishing the relationship between perception ; ; ; , prediction ; ; , and planning ; ; ; in autonomous driving is crucial for a comprehensive understanding of the field.

Conventional autonomous driving models rely on the motion of the ego vehicle instances to model the development of scenes, unable to develop a profound understanding of scene perception and vehicle motion control comparable to human understanding.
The emergence of world models offers new possibilities for a deeper understanding of the comprehensive relationship between autonomous driving scenes and vehicle motion.
Based on strong image pretrained models, image-based world models ; can generate high-quality driving-scene images with conditions of 3D bounding boxes.
OccWorld  further learns a world model in the 3D occupancy space, which can be better leveraged for 3D reasoning for autonomous driving. However, most existing methods adopt an autoregressive framework to model the dynamics (e.g., image tokens, bounding boxes, occupancy) of a 3D scene, which hinders their ability to efficiently produce long-term video sequences.

Figure: Figure 1: Comparisons with existing methods. It can comprehend the intricate relationship between scenes and trajectories and generate long-term, physically consistent 4D occupancy.
Refer to caption: x1.png

To address this, we propose a 4D world model OccSora to directly generate spatial-temporal representations with diffusion models as shown in Figure 1, motivated by OpenAI’s 2D video generation model Sora .
To accurately understand and represent 4D scenes, we design 4D scene discretization to capture the dynamic characteristics of scenes and propose a diffusion-based world model to achieve controllable scene generation following physical laws. Specifically, in the 4D occupancy scene tokenizer, we focus on extracting and compressing real 4D scenes to establish an understanding of the world model environment. In the diffusion-based world model, we employ multidimensional diffusion techniques to propagate accurate spatiotemporal 4D information and realize trajectory-controllable scene generation by incorporating real ego car trajectories as condition, thereby achieving a deeper understanding between autonomous driving scenes and vehicle motion control. Through training and testing, OccSora can generate autonomous driving 4D occupancy scenes that adhere to physical logic and achieve controllable scene generation based on different trajectories. The proposed autonomous driving 4D world model opens up new possibilities for understanding dynamic scene changes in autonomous driving and the physical world.

## 2 Related Work

3D Occupancy Prediction. 3D occupancy focuses on partitioning space into voxels and assigning specific semantic types to each voxel. It is considered a crucial means of representing real-world scenes, following 3D object detection ; ; and Bird’s Eye View (BEV) perception ; ; ; , for autonomous driving perception tasks. Early research on this task primarily focused on semantically classifying discrete points from LiDAR ; ; ; . In fact, due to the camera containing semantic information far exceeding that of LiDAR and their low cost. Thus, utilizing images for depth estimation or employing end-to-end methods for 3D scene perception research is currently the mainstream approach ; ; ; . Considering the advantages of multi-sensor systems, some studies research multi-modal fusion for 3D occupancy prediction ; .

In addition to utilizing typical sensor devices for 3D occupancy prediction, some studies focus on other tasks involving occupancy. For instance, OccWorld proposes a spatiotemporal generative transformer to predict subsequent scene tokens and the vehicle token, thereby predicting future occupancy and vehicle trajectory. On the other hand, GenOcc utilizes generative models to accomplish occupancy prediction. DriveWorld introduces a world-model-based framework for learning in autonomous driving from 2D images and videos, addressing tasks such as 3D object detection, online map creation, and occupancy prediction.
Although progress has been made in 3D occupancy prediction and continuous 4D prediction, the scope of these studies remains limited.
They usually use autoregressive models in conjunction with scene information from preceding frames to carry out subsequent occupancy tasks, thereby necessitating prior scene or 3D bounding box inputs. Consequently, they lack a genuine understanding of the fundamental relationships between scene and motion, and therefore do not constitute world models conditioned on actions.

Figure: Figure 2: The pipeline of OccSora. The 4D occupancy scene tokenizer achieves compression and restoration of real information. The compressed information and vehicle trajectories are simultaneously used as inputs for the diffusion-based world model. After training, the diffusion-based world model utilizes random noise and arbitrary trajectories to generate controllable tokens, which are then decoded into 4D occupancy maps in the 4D occupancy scene tokenizer stage.
Refer to caption: x2.png

Generative Model.
Generative models have garnered widespread attention recently due to their powerful capabilities.
By learning the probability distribution of data, generative models can train models capable of generating new samples. From the emergence of Generative Adversarial Networks (GAN) to the recent advent of diffusion models like Variational Autoencoders (VAE) , the tasks of generative models have gradually expanded from initial image generation tasks to in-depth studies on videos . Tasks such as image generation based on the DIT model delve into and utilize their generative capabilities. The Sora video generation model further demonstrates the ability to produce high-quality videos with realistic transitions between frames in continuous scenes.

Similarly, in the field of autonomous driving, controllable image generation can provide various driving scenarios to serve perception, planning, control, and decision-making tasks. For instance, MagicDriver generates videos depicting various weather scenarios by learning from videos of autonomous driving vehicles and incorporating labels such as object detection boxes and maps. DriveDreamer proposes a world model that is entirely derived from real-world driving scenes, enabling a deep understanding of structured traffic constraints and thereby achieving precise and controllable video generation. However, for autonomous driving scenarios, obtaining the 3D occupancy of scenes is more important compared to 2D information ; ; . Some studies ; propose a three-dimensional diffusion model suitable for generating outdoor real scenes, which, by utilizing diffusion methods, accomplishes scalable seamless scene generation tasks. While some previous studies have generated 2D static images and extended them to the temporal dimension through autoregression, and others have achieved static generation of 3D occupancy scenes, both the 2D images generated based on 3D object bounding boxes and the static large-scale scenes are difficult to directly apply to autonomous driving tasks ; . In contrast, our proposed OccSora establishes a dynamic 4D occupancy world model that adapts to scene changes with vehicle trajectories, without the need for any prior object detection boxes or scene information, representing the first generative 4D occupancy world model for autonomous driving.

## 3 Proposed Approach

### 3.1 World Model for Autonomous Driving

4D occupancy can comprehensively capture the structural, semantic, and temporal information of a 3D scene and effectively facilitate weak supervision or self-supervised learning, which can be applied to visual, LiDAR, or multimodal tasks.
Based on these principles, we represent the world model $\chi$ as 4D occupancy $R$. Figure 2 illustrates the overall framework of OccSora. We constructed a 4D occupancy scene tokenizer to compress real 4D occupancy $R_{in}\in\mathbb{R}^{B\times D\times H\times W\times T}$ in both the temporal $T$ and spatial $D\times H\times W$ dimensions, capturing the relationships and evolution patterns in 4D autonomous driving scenes. This results in compressed high-level tokens $R_{mi}\in\mathbb{R}^{B\times c\times h\times w\times t}$ and reconstructed 4D occupancy data $R_{o}\in\mathbb{R}^{B\times D\times H\times W\times T}$. We designed a diffusion-based world model that uses trajectory information $R_{tr}\in\mathbb{R}^{B\times T\times 2}$ as control units, training them supervised by the compressed tokens $R_{mi}$ to generate high-dimensional scene representation tokens $T_{o}\in\mathbb{R}^{B\times c\times h\times w\times t}$. They are then decoded by the 4D occupancy scene tokenizer to consistent and dynamically controllable $R_{o}$.

### 3.2 4D Occupancy Scene Tokenizer

The goal of 4D occupancy prediction is to determine the semantic type at specific locations over time. We discretize and encode the real 4D occupancy scene $R_{in}$ into an intermediate latent space $R_{mi}$ to obtain a true representation of the 4D occupancy scene, as shown in Figure 3. The formula is as follows: $R_{mi}=\zeta_{token}\left\{\tau_{en}\left(R_{in}\right)\right\}.$ Here, $\zeta_{token}$ represents the encoded codebook, and $\tau_{en}$ denotes the designed 3D encoder network and category embedding. This 3D occupancy representation divides the 3D space around the vehicle into voxels $r^{T}=N\in\mathbb{R}^{H\times W\times D}$, where each voxel position is assigned a type label $N$, indicating whether it is occupied and the semantics of the object occupying it. Unlike traditional methods, we incorporate and compress temporal information within the same scene, reshaping the tensor to $R_{in}$. This approach allows for unified learning of both spatial and temporal evolution patterns and the physical relationships of real scenes, compared to previous autoregressive methods. After passing through the $\tau_{en}$ 3D encoder network with category embedding and the $\zeta_{token}$ encoded codebook, the tensor is transformed into $R_{mi}$ represents the potential spaces. This reshaping ensures a comprehensive representation of the temporal dynamics of 4D occupancy.

Figure: Figure 3: The structure of the 4D occupancy scene tokenizer. The proposed method encodes and compresses 4D scenes to extract high-dimensional features, which are then decoded to retrieve the spatiotemporal physical characteristics of the scenes.
Refer to caption: x3.png

Category Embedding and Tokenizer. To accurately capture the spatial information of the original parameters, we first perform an embedding operation on the input $R_{in}$. We assign a learnable category embedding $b\in\mathbb{R}^{c^{\prime}}$ for each category in $R_{in}$ to label the categories of continuous 3D occupancy scenes. The position information is embedded as tokens that represent the categories. Then, these embeddings are concatenated along the feature dimension. To facilitate subsequent 3D encoding with compression in specific dimensions, we further reshape $R_{in}$ into $R_{in}^{\prime}\in\mathbb{R}^{B\times(Dc^{\prime})\times T\times H\times W}$.

3D Video Encoder. To effectively learn discrete latent tokens, we further performed downsampling on the embedded positional information of the 4D occupancy $R_{in}^{\prime}$ to extract high-dimensional features. The designed encoder architecture comprises a series of 3D downsampling convolutional layers, which perform 3D downsampling in both the time dimension (T) and spatial dimensions (H × W), increasing the fusion dimension to $D\times c^{\prime}$. We initially downscaled the input $R_{in}^{\prime}$ three times to obtain $R_{in}^{\prime\prime}\in\mathbb{R}^{B\times(8\times Dc^{\prime})\times\frac{T}
{8}\times\frac{H}{8}\times\frac{W}{8}}$, and introduced dropout layers after the feedforward and attention block layers for regularization. Considering the relationships between consecutive frames, we introduced cross-channel attention after downsampling, segmenting $R_{in}^{\prime\prime}$ along the $8\times Dc^{\prime}$ dimension and then performing cross-channel attention between the segmented parts. This operation enhanced the model ability to capture relationships between features along different axes, and subsequently reshaped them back to the original shape to obtain the output tensor $R_{mi}$.

Coodbook and Training Objective. To achieve a more condensed representation, we simultaneously learn a codebook $\zeta_{token}\in\mathbb{R}^{N\times D}$ containing N codes. Each code $b\in\mathbb{R}^{c^{\prime}}$ in the codebook encodes a high-level concept of the scene, such as whether the corresponding position is occupied by a car. $\zeta_{token}$ represents the encoded codebook. We quantize each spatial feature $\widehat{R_{mi}^{\left(ij\right)}}$ in $\widehat{R_{mi}}$ by mapping it to the nearest code $N(\widehat{R_{mi}^{\left(ij\right)}},B)$:

$$ $R_{mi}^{\left(ij\right)}=N(\widehat{R_{mi}^{\left(ij\right)}},\zeta_{token})= \min_{b\in\zeta_{token}}||\widehat{R_{mi}^{\left(ij\right)}}-b||_{2},$ (1) $$

where $||\cdot||_{2}$ represents the L2 norm. Subsequently, we integrate the quantized features $\widehat{R_{mi}^{\left(ij\right)}}$ to obtain the final scene representation $R_{mi}$.

3D Video Decoder. To reconstruct $R_{o}$ from the learned scene representation $R_{mi}$, we design a decoder consisting of 3D deconvolution layers. In contrast to the encoder, the decoder architecture includes cross-channel attention, residual blocks, and a series of 3D convolutions, enabling upsampling in both temporal and spatial dimensions. This gradual upsampling process transforms $R_{mi}$ to its original occupancy resolution $R_{o}$. The decoder then splits the result along the channel dimension to reconstruct the temporal dimension, yielding occupancy values for each voxel.
During training, we accomplished the training of the encoder, decoder parameters, and the encoding codebook. The designed network enables us to simultaneously encode the input 4D occupancy information and compress it into multiple tokens, thereby learning the physical correlations of world models under spatiotemporal fusion.
Additionally, we restore the information during the decoding process.

### 3.3 Diffusion-based World Model

Figure: Figure 4: The structure of the diffusion-based world model. The model involves utilizing the optimal codebook obtained from training the 4D occupancy scene tokenizer to convert 4D occupancy into a sequence of tokens. These tokens, along with the ego vehicle trajectory and random noise, are then combined as input for denoising training to acquire the generated token.
Refer to caption: x4.png

Inspired by the diffusion method , we use scene tokens $R_{mi}$ containing spatiotemporal information features as inputs for the generative model. Additionally, we conduct denoising training and trajectory-controllable generation tasks under the control of vehicle trajectories $R_{tr}$, as shown in Figure 4.

Token Embedding. To efficiently and accurately utilize the transformer , we flatten the input data tokens $R_{mi}$ into $R_{\text{re}}$. Simultaneously, considering the significance of positional information for spatiotemporal compression, we perform positional embedding on the input. We design the following function, which utilizes sin and cos functions to encode positional indices:

$$ $R_{re}^{\left(\mathrm{emb}\right)}=\mathrm{emb}_{\mathrm{i}}^{\mathrm{d}}+R_{ re},R_{re}\in\mathbb{R}^{\mathrm{B}\times\mathrm{c}\times\left(\mathrm{hwt} \right)}.$ (2) $$

It operates on two main parameters: $C$, representing the embedding output dimensionality of each position, and $i=hwt$, representing the number of tokens enumerating the positions to be encoded. The resulting output follows a matrix structure of dimensions $C\times i$, and $emb$ constructs the positional embedding representation using sin and cos functions. These embeddings encapsulate the positional attributes of the tokens, enhancing the model understanding of positions within the input. We add the positional encoding $E_{mb}$ to the input $R_{re}$, yielding $R_{re}^{\left(\mathrm{emb}\right)}$, which represents the tokens after positional encoding.

Trajectory Conditioning Embedding. The transformation relationship between scenes and trajectories is a crucial aspect of autonomous driving. Generating diverse 4D occupancy scenes that align with control trajectories is essential. Therefore, we use the ego vehicle trajectory $T_{r}$ as input to generate controllable 4D occupancy. Firstly, the ego vehicle trajectory $T_{r}\in\mathbb{R}^{B\times t\times 2}$ is used as one of the control inputs, where $t$ denotes the continuous time dimension, and the third dimension represents the vehicle positions along the $x$ and $y$ axes of the absolute coordinate system. To achieve trajectory embedding and encoding, we reshape the vehicle trajectory to $T_{r}\in\mathbb{R}^{B\times\left(t\times 2\right)}$ and learn and encode it as follows:

$$ $g=\nu\left(t\right)+\delta\left(T_{r}\right),\delta\left(T_{r}\right)\in \mathbb{R}^{\mathrm{B}\times\mathrm{c}\times\left(\mathrm{hwt}\right)},\nu\in \mathbb{R}^{\mathrm{B}\times\mathrm{c}\times\left(\mathrm{hwt}\right)},$ (3) $$

where $\nu\in\mathbb{R}^{\mathrm{B}\times\mathrm{c}\times\left(\mathrm{hwt}\right)}$ represents the time step embedding, and $\delta$ denotes the Multilayer Perceptron (MLP) network that extracts trajectory information. It is then embedded $g$ into the input sequence of the diffusion transformer and processed together with the token information $R_{re}^{\left(\mathrm{emb}\right)}$.

Diffusion Transformer. We developed a diffusion-based world model to learn from and generate within the latent space $R_{mi}$, while integrating trajectory labels $T_{r}$ and denoising time steps $\nu$ as control conditions. In the model diffusion learning process, we constructed a forward noise process that gradually introduces noise to the latent space $R_{mi}$:
$q\left(R_{\mathrm{re}}^{g}|R_{\mathrm{re}}\right)={N}\left(R_{\mathrm{re}}^{g}
;\sqrt{\overline{\sigma^{g}}}R_{\mathrm{re}},\left(1-\overline{\sigma^{g}}
\right)I\right),$ where the constant $g$ represents the embedding of trajectories and time steps. Utilizing the reparameterization trick, we can sample: $R_{\mathrm{re}}^{g}=\sqrt{\overline{\sigma^{g}}}R_{\mathrm{re}}+\sqrt{1-
\overline{\sigma^{g}}}\epsilon^{g}$, where $\epsilon^{g}\sim{N}\left(0,I\right)$. The 4D occupancy diffusion model is trained to learn the reverse propagation process. To invert the forward process corruption:

$$ $p_{\theta}\left(R_{\mathrm{re}}^{g-1}|R_{\mathrm{re}}^{g}\right)={N}\left(\mu_ {\theta}\left(R_{\mathrm{re}}^{g}\right),\Sigma_{\theta}\left(R_{\mathrm{re}}^ {g}\right)\right),$ (4) $$

where neural networks predict the statistical properties of $p$. The reverse process model is trained with the variational lower bound of $x_{0}$, which simplifies to: ${L}\left(\theta\right)=-p\left(R_{\mathrm{re}}^{0}|R_{\mathrm{re}}^{1}\right)+
\sum{{}_{g}{D}_{KL}\left(q^{*}\left(R_{\mathrm{re}}^{g-1}|R_{\mathrm{re}}^{g},
R_{\mathrm{re}}^{0}\right)\right)||p_{\theta}\left(R_{\mathrm{re}}^{g-1}|R_{
\mathrm{re}}^{g}\right)},$ which excluding irrelevant additional terms during training. As both $q$ and $p$ are gaussian distributions, the Kullback-Leibler (KL) divergence can be evaluated using the means and covariances of the two distributions. By reparameterizing as a noise prediction network, the model can be trained using the simple mean squared error between the predicted noise $\hat{R}_{re}^{g}$ and the sampled gaussian noise $R_{re}^{g}$: ${L}_{simple}(\theta)=\frac{1}{2}(\hat{R}_{re}^{g}-R_{re}^{g})^{2}.$ However, to train the diffusion model with learned reverse process covariance, the full KL divergence term needs to be optimized. We follow diffusion models approach : train first with ${L}_{simple}(\theta)$, then with the full ${L}$. Once $p$ is trained, new token can be sampled by initializing $R_{re}^{g}\sim N(0,I)$ and sampling $R_{re}^{g-1}\sim p(R_{re}^{g-1}|R_{re}^{g})$ using the reparameterization trick.

Overall, tokens $R_{mi}$ processed in the initial stage as $R_{re}$ are passed to a series of transformer blocks for further refinement. These blocks effectively capture the relationships between trajectory information and tokens. Regarding noisy image input processing, the diffusion transformer employs specific attention mechanisms and loss functions to minimize the impact of noise on model performance, ensuring robust operation in noisy environments. To incorporate trajectory labels $T_{r}$ and denoising time steps $\nu$ as additional control conditions, we feed them as supplementary inputs alongside token embeddings into the transformer blocks. This enables the model to dynamically adjust its processing based on these conditions, thereby better adapting to various trajectory control requirements. In the end, the trained diffusion-based world model successfully transforms pure noise and trajectory labels $T_{r}$ into $T_{o}\in\mathbb{R}^{B\times c\times h\times w\times t}$, which are eventually decoded into $R_{o}$ through the 3D decoder.

## 4 Experiments

As a 4D occupancy world model in the field of autonomous driving, OccSora offers a deeper understanding of the relationship between autonomous driving scenes and vehicle trajectories without requiring any input of 3D bounding boxes, maps, or historical information. It can construct a long-time sequence world model that adheres to physical laws. We have conducted a series of quantitative experiments and visualizations to illustrate this.

### 4.1 Implementation Details

We conducted experiments on the widely used nuScenes-Occupancy dataset , which is currently one of the most mainstream and standard datasets, supporting many well-known research studies ; . For the OccSora model, we applied three rounds of compression to 32 consecutive frames and increased its channel dimension to 128. Subsequently, we conducted further comparative and ablation experiments under different components and trajectory scenarios. We trained using the AdamW optimizer with an initial learning rate set to $1\times 10^{-5}$ and a weight decay of 0.01. Using 8 NVIDIA GeForce A100 GPUs, we set a batch size of 2 per GPU. For the training of the 4D occupancy scene tokenizer, we needed about 42GB of memory per GPU to train for 150 epochs, which took 50.6 hours. For the diffusion-based world model, we needed about 47GB of memory per GPU to train for 1,200,000 steps, which took 108 hours.

**Table 1: The quantitative analysis of 4D occupancy reconstruction. Despite a compression rate 32 times greater than OccWorld, OccSora maintains over half of its reconstruction accuracy compared to OccWorld.**
| Method | Ratio | IoU | mIoU | Others | barrier | bicycle | bus | car | const. veh. | motorcycle | pedestrian | traffic cone | trailer | truck | drive. suf. | other flat | sidewalk | terrain | man made | vegetation |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| OccWorld | 16 | 62.2 | 65.7 | 45.0 | 72.2 | 69.6 | 68.2 | 69.4 | 44.4 | 70.7 | 74.8 | 67.6 | 54.1 | 65.4 | 82.7 | 78.4 | 69.7 | 66.4 | 52.8 | 43.7 |
| OccSora | 512 | 37.0 | 27.4 | 11.7 | 22.6 | 0.0 | 34.6 | 29.0 | 16.6 | 8.7 | 11.5 | 3.5 | 20.1 | 29.0 | 61.3 | 38.7 | 36.5 | 31.1 | 12.0 | 18.4 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Figure: Figure 5: Visualization of reconstruction of the 4D occupancy scene tokenizer.
Refer to caption: x5.png

**Table 2: Comparisons of OccSora with other models in its generation capability. To the best of our knowledge, we are the first 4D occupancy generation model. Therefore, we only provide comparisons with other generative models on different datasets and data formats.**
| Method | Type | Dimension | Dataset | FID |
| --- | --- | --- | --- | --- |
| DiT | Image | 2D | ImageNet | 12.03 |
| MagicDriver | Video | 3D | nuScenes | 14.46 |
| DriveDreamer | Video | 3D | nuScenes | 14.9 |
| DriveGAN | Video | 3D | nuScenes | 27.8 |
| SemCity | Occupancy | 3D | KITTI | 40.63 |
| OccSora | Occupancy Video | 4D | nuScenes | 8.348 |

Figure: Figure 6: The visualization of the progressive generation of accurate scenes as the model undergoes iterative training.
Refer to caption: x6.png

### 4.2 4D Occupancy Reconstruction

The compression and reconstruction of 4D occupancy are essential for learning the latent spatiotemporal correlations and features necessary for image generation. Unlike traditional models for video and image processing, OccSora operates one dimension higher than occupancy for single frames and two dimensions higher than images. Therefore, achieving efficient compression and accurate reconstruction is paramount. Figure 5 depicts the ground truth and reconstruction of the occupancy. We also conducted a quantitative analysis of 4D occupancy reconstruction, as shown in Table 1. The table indicates that even with OccSora achieving a compression ratio 32 times greater than that of OccWorld, it still maintains nearly 50% mIoU of the original OccWorld model. This unified temporal compression effectively captures the dynamic changes of various elements, improving long-sequence modeling capabilities compared to progressive autoregressive methods.

### 4.3 4D Occupancy Generation

In the diffusion-based world model for the 4D occupancy generation task, we used tokens generated by the OccSora model, trained with 32 frames, as input for our generation experiments. In Figure 6, we present the visual results of across training iterations, from 10,000 to 1,200,000 steps. These visual results indicate that as the number of training iterations increases, the accuracy of the OccSora model continuously improves, demonstrating the generation of coherent scenes.

Figure: Figure 7: 4D occupancy generation under different input trajectories. From top to bottom, there is go straight, turning right, and motionless, with each scene generation corresponding to the trajectory, ensuring logical coherence and continuity.
Refer to caption: x7.png

Figure: Figure 8: Generating diverse continuous scenes under trajectory control. The generated scenes exhibit diversity while maintaining the stability of the original trajectory control.
Refer to caption: x8.png

**Table 3: Results of ablative evaluation on different components. We quantitatively evaluated the impact of different compression rates, components, and channel dimensions on the reconstruction and generation results through controlled variables.**
| Input Size $R_{in}$ | Token Size $R_{mi}$ | Channel | Class | T embed. | Trajectory | IoU | mIoU | FID |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 32x200x200 | 128x4x25x25 | 8 | ✓ | ✓ | ✓ | 37.03 | 27.42 | 8.34 |
| 32x200x200 | 128x4x25x25 | 8 | ✓ | ✗ | ✓ | 37.03 | 27.42 | 87.26 |
| 32x200x200 | 128x4x25x25 | 8 | ✓ | ✓ | ✗ | 37.03 | 27.42 | 17.48 |
| 32x200x200 | 128x4x25x25 | 4 | ✓ | ✓ | ✓ | 29.67 | 23.21 | 34.24 |
| 32x200x200 | 128x8x50x50 | 8 | ✓ | ✓ | ✓ | 32.91 | 24.4 | 72.32 |
| 12x200x200 | 64x3x50x50 | 8 | ✓ | ✓ | ✓ | 26.73 | 14.12 | 187.78 |
| 12x200x200 | 64x3x25x25 | 8 | ✓ | ✓ | ✓ | 22.42 | 9.27 | 270.23 |
| 12x200x200 | 32x3x25x25 | 8 | ✓ | ✓ | ✓ | 13.60 | 3.85 | 465.18 |

We compared and quantitatively evaluated our proposed OccSora model against other generation models. As the first 4D occupancy world model for autonomous driving, we only compared it against conventional image generation, 2D video generation, and static 3D occupancy scene generation methods. As shown in Table 2, our model achieves similar performance in terms of the Fréchet Inception Distance (FID) , demonstrating the effectiveness of the proposed method.

Trajectory Video Generation. OccSora has the capability to generate various dynamic scenes based on different input trajectories, thus learning the relationship between ego vehicle trajectories and scene evolution in autonomous driving. As shown in Figure 7, we input different vehicle trajectory motion patterns into the model, demonstrating the 4D occupancy for go straight, turning right, and motionless. We conducted experiments at different scales for generating trajectories, revealing that the FID score is lowest for stationary scenes and higher for curved scene, indicating the complexity of continuously modeling curved motion scenes and the simplicity of modeling stationary scenes.

**Table 4: The quantitative analysis of different scales regarding denoising steps and denoising rates. Denoising steps have a relatively minor impact on the model, whereas denoising rates and model scales significantly affect the quality of the generated outputs.**
| Step | Input Size $R_{in}$ | Token Size $R_{mi}$ | FID |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 10% | 30% | 50% | 70% | 90% | 100% |  |  |  |
| 10 | 32x200x200 | 128x4x25x25 | 49863 | 34927 | 17630 | 339 | 42 | 9.1 |
| 100 | 32x200x200 | 128x4x25x25 | 53297 | 29521 | 19471 | 1084 | 72 | 10.08 |
| 1000 | 32x200x200 | 128x4x25x25 | 32171 | 10284 | 5924 | 591 | 17 | 8.94 |
| 10 | 12x200x200 | 64x3x50x50 | 71293 | 54625 | 5644 | 7416 | 742 | 431 |
| 100 | 12x200x200 | 64x3x50x50 | 81274 | 53431 | 45346 | 3161 | 456 | 446 |
| 1000 | 12x200x200 | 64x3x50x50 | 43631 | 33415 | 17431 | 4366 | 379 | 353 |

Figure: Figure 9: Effect of the denoising ratios under different trajectories or denoising steps. Denoising steps and trajectories have minor impacts while denoising ratios have a significant effect.
Refer to caption: x9.png

Scene Video Generation. Diversity in scenes is crucial under reasonable trajectory control. We tested the reconstruction of 4D occupancy scenes for different scenarios under three trajectories to verify the generalization performance of generating scenes under controllable trajectories. In Figure 8, the left and right parts respectively demonstrate the capability to generate different scenes under the same trajectory. In the reconstructed scenes, surrounding trees and road environments exhibit random variations while still maintaining the logic of the original trajectory, showcasing the ability to maintain robustness in generating scenes corresponding to the original trajectory and its generalization across different scenarios.

### 4.4 Ablation and Analysis

Analysis of the Tokenizer and Embeddings.
We conducted an ablation of the proposed components including different compression scales, the number of class tokenizer discretizations, time-step embeddings, and vehicle trajectory embeddings, as shown in Table 3.
When the number of class tokenizer discretizations was reduced from 8 to 4, the reconstruction accuracy dropped by approximately 18%. The FID score also declined after removing the time-step embeddings. Without position embeddings, the generated scenes lacked motion control and displayed almost linear movement patterns influenced by the data distribution. Additionally, at lower compression ratios, although the reconstruction performance was better compared to higher compression ratios, the lack of higher-dimensional feature correlations prevented the generation of effective scenes.

Analysis of the Generation Steps. The total number of denoising steps and the denoising rate can affect the generation quality to some extent. As shown in Figure 9, as the denoising rate increases, the generated scenes become progressively clearer. According to the quantitative results in Table 4, increasing the total number of denoising steps can improve generation accuracy to a certain extent. However, the generation quality is much more significantly influenced by the token size and the number of channels than by the total number of denoising steps.

## 5 Conclusion and Limitations

In this paper, we have introduced a framework for generating 4D occupancy to simulate 3D world development in autonomous driving. Using a 4D scene tokenizer, we obtain compact representations for input and achieve high-quality reconstruction for long-sequence occupancy videos. Then, we learn a diffusion transformer on the spatiotemporal representations and generate 4D occupancy conditioned on a trajectory prompt. Through experiments on the nuScenes dataset, we demonstrate accurate scene evolution. In the future, we will investigate more refined 4D occupancy world models and explore the possibilities of end-to-end autonomous driving under closed-loop settings.

Limitations. The advantage of a 4D occupancy world model lies in establishing an understanding of the relationship between scenes and motion. However, due to limitations in the granularity of voxel data, we cannot construct more finely detailed 4D scenes.
The generative results also demonstrate inconsistent details for moving objects, possibly due to the small size of the training data.

## Appendix A Appendix

### A.1 More Visualizations

To provide results for longer time series, we present the generated scenes under different ego vehicle trajectory controls, namely Go Straight, Turning Right, Motionless, and Accelerate in Figure 10. Additionally, we also showcase the equivalent control methods under different scenes in Figure 11.

Figure: Figure 10: The ability to generate long time-series 4D occupancy under different trajectory controls. From top to bottom, we present long-term continuous scenes generated under four types of ego vehicle trajectories: Go Straight, Turning Right, Motionless, and Accelerate.
Refer to caption: x10.png

Figure: Figure 11: The generalization ability to generate different scenes under fixed ego vehicle trajectories. From top to bottom, we show the capabilities of generating different scenes under the four vehicle trajectories: Go Straight, Turning Right, Motionless, and Accelerate.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See the weaknesses above
soundness: 3
strengths: 1.The paper is well written and organized. It’s easy to understand.
2.The authors provide abundant visualization results to show the effectiveness.
summary: The paper introduces a 4D occupancy generation model called OccSora, designed to simulate the evolution of environments in autonomous driving scenarios. It proposes a 4D scene encoder to produce a compact spatiotemporal representation and uses a diffusion model to generate 4D occupancy sequences based on given trajectory prompts. OccSora addresses the inefficiency of existing autoregressive models in long-term temporal simulation, providing more physically consistent simulation support for decision-making in autonomous driving through trajectory-aware 4D generation. OccSora achieves good results on nuScenes dataset.
weaknesses: 1. The novelty is limited. Using diffusion to generate occupancy is widely studied. And the contribution seems incremental compared to previous works in the field of occupancy prediction.
2.  It might be beneficial to conduct experiments on additional datasets, such as Waymo, to enhance the persuasiveness of the paper. 
3. The text in Figure 3 is too small and unclear.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Refer to Weaknesses.
soundness: 3
strengths: This paper presents OccSora, a pioneering diffusion-based 4D occupancy generation model that utilizes a 4D scene tokenizer to capture compact spatiotemporal representations, improving the modeling of long-term temporal dynamics in autonomous driving. The research is comprehensive, featuring extensive experiments on the nuScenes dataset that showcase OccSora's ability to generate realistic 16-second videos with stable 3D layouts.  The paper is clearly structured and well-written, offering thorough explanations of the model architecture, training procedure, and evaluation metrics. OccSora’s capability to generate trajectory-aware 4D occupancy scenes makes it a valuable asset for decision-making in autonomous driving, with the potential to enhance safety and efficiency.
summary: This paper introduces OccSora, a diffusion-based model for 4D occupancy generation in autonomous driving. Using a 4D scene tokenizer, OccSora captures compact and discrete spatiotemporal representations, allowing for the high-quality reconstruction of long-sequence occupancy videos. A diffusion transformer is subsequently trained on these representations to generate 4D occupancy, conditioned on a trajectory prompt. Experimental results on the nuScenes dataset with Occ3D occupancy annotations show that OccSora can produce 16-second videos with realistic 3D layouts and temporal coherence, demonstrating its understanding of spatial and temporal patterns in driving scenes. This method shows potential as a world simulator for decision-making in autonomous driving, addressing the inefficiencies in modeling long-term temporal evolution found in autoregressive approaches.
weaknesses: The writing of this paper should be enhanced. For example,

1）Some details in the methods section (Section 3) could be presented with greater clarity. For ‘Category embedding and Tokenizer’, occupancy is represented by Rin in the text, while in Figure 3, occupancy is represented by Ro.

2）If Figures 2, Figure 3 and Figure 4 contain elements corresponding to Rin, Rmi, Ro, and Rtr, please ensure these are labeled directly within the figures.

3）Please clearly indicate the dimensional changes between features within the figure to enhance reader comprehension.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: The reviewer has identified five major concerns and would like the authors' responses to these points. Please answer each concern in the rebuttal stage. The reviewer will respond according to the authors' rebuttal in the discussion phase.
soundness: 2
strengths: This paper proposes a trajectory-aware 4D occupancy generation model, which can comprehend the trajectories of ego car and realize trajectory-controllable scene generation. The model uses a 4D scene tokenizer to create compact, discrete spatial-temporal representations of 4D occupancy inputs, achieving high-quality reconstructions for long-sequence occupancy videos. The proposed OccSora can generate long-term occupancy videos up to 16 seconds.
summary: The paper introduces a trajectory-aware 4D occupancy generation model capable of understanding ego car trajectories and enabling trajectory-controllable scene generation.
weaknesses: 1. This paper claims in the abstract and introduction that existing autoregressive methods "suffer from inefficiency to model long-term temporal evolutions". It is unclear how the authors address this, as the multiple denoising steps of the diffusion model can also be time-consuming. I suggest that the authors provide some **evidences to demonstrate that the proposed architecture is more efficient**, such as runtime comparisons or theoretical complexity analyses between the proposed approach and existing autoregressive methods.
2. The experiment on the 4D occupancy prediction is not convincing. The metrics indicate that OccSora significantly outperforms OccWorld; however, the **generation quality** in qualitative results is not good enough. It is beneficial to provide side-by-side visual comparisons between OccSora and baseline methods, highlighting specific areas where the generation quality differs.
3. It is unclear **how 4D occupancy prediction is conducted** in Fig. 5 and Tab. 10, as there appears to be no frame condition in the proposed architecture. I suggest that the authors provide a more detailed explanation of how the model handles frame conditioning for 4D occupancy prediction in the proposed architecture.
4. As shown in Fig.12, the model can generate different scenes with different motion conditions. However, their first frames are different. As far as the reviewer knows, if the authors want to **claim the model is controllable**, the model should be able to generate different future predictions with the same initial frame(s) using different trajectories. I recommend that the authors provide additional experiments or qualitative results to demonstrate this .
5. Why are the IoU and mIoU values at 0s in Table 5 different from the reconstruction results in Table 1? Both seem to measure reconstruction performance.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In Table 2, it's unclear why increasing the resolution (from $ 128 \times 4 \times 25 \times 25 $ to $ 128 \times 8 \times 50 \times 50 $) improves the reconstruction IoU and mIoU, but results in a decrease in generation FID. This discrepancy needs further explanation.
2. The figures showing occupancy data throughout the paper are quite small, and the resolution is low, making it difficult to see details. The authors should consider placing fewer samples in each row and enlarging the figures for better visibility.
soundness: 3
strengths: 1. This paper is well-organized, presenting a clear flow making it easy to read.
2. This paper proposes a generative 4D occupancy world model designed for autonomous driving, along with a novel generation task for occupancy data.
3. The experiments provide an evaluation of the model's capabilities and performance across various scenarios.
summary: This paper proposes OccSora, a diffusion-based 4D occupancy generation model designed for autonomous driving. It utilizes a 4D scene tokenizer to get spatial-temporal representations, allowing for high-quality reconstruction of long-term occupancy videos. The model generates 4D occupancy data based on trajectory prompts, generating videos with 3D layouts and temporal consistency. Experiments demonstrate OccSora's potential as a world simulator for autonomous driving.
weaknesses: 1. The quality of the turning cases shown in Figure 8 and Figure 13 appears to be quite poor. The motion trend is barely visible, and the scene structure beside the road becomes corrupted in the later frames. This undermines the claim of trajectory awareness. I suspect this issue is related to the imbalanced data distribution in the nuScenes dataset, which could present an interesting challenge. Unfortunately, the paper offers no further analysis or solution regarding this problem.
2. The authors spent a lot of space discussing the diffusion transformer (lines 269–290). However, much of this appears to replicate the contribution of the original DIT paper. This section could be shortened or moved to a preliminary section instead.
3. The mathematical notation throughout the paper is quite disorganized. Some of the symbols are unusual, and the authors should consider using more standard notation (e.g., what is "mi" in $ R_{mi} $?). The excessive use of superscripts and subscripts makes it difficult to follow. Additionally, the authors should avoid using unnecessary symbols for unimportant variables (e.g., $ x $ in line 246). There are also several instances of misused symbols. For example:
  1. The symbol $ N $ is used multiple times for different meanings, such as a label (line 176), nearest code operation (line 205), and Gaussian noise (line 271).
  2. In line 254, the positional encoding notation is inconsistent with Equation (2).
  3. In Equation (3), the $ t $ in $ v(t) $ should represent the diffusion timestep, but it is confused with the occupancy timestep $ t $.
  4. In line 265, $ v $ is referred to as the waypoint timestep embedding, but it is suddenly called the denoising timestep in line 269.

4. There are several grammar issues and typos:
  1. In line 181, "represents" should be "representing."
  2. In line 197, "model ability" should be "model's ability."
  3. In line 253, "model understanding" should be "model's understanding."
  4. In line 266, it should read: " $ g $ is then embedded into the input sequence ..."
  5. In line 280, "occuancy" should be "occupancy."
  6. In line 525, "reprort" should be "report."

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 1
questions: 1. **The main text of the paper exceeds the maximum page limit of 10 pages**, as per the ICLR 2025 guidelines. Please check the details at ICLR Call for Papers.
2. The authors should consider using `\citep{}` in the paper for better formatting. Refer to the ICLR LaTeX template for the appropriate usage and differences between citation commands.
3. In the caption of Table 1, the output dimension for the "512x" setting should be listed as $ 25 \times 25 \times 4 $.
4. In Figure 6, the model name should be updated to **OccSora-base**.
soundness: 3
strengths: 1. The paper is well-organized, making it easy to follow.
2. This paper introduces the first generative 4D occupancy world model for autonomous driving and proposes a new generation task for occupancy data, which is a contribution to the community.
3. The design of the 4D tokenizer (Section 3.2) is particularly novel. The experiments demonstrate its effectiveness in preserving 3D occupancy geometry with temporal consistency, even under high compression rates.
4. The ablation study is thorough and comprehensive.
summary: This paper introduces **OccSora**, a 4D occupancy generation model designed to capture the evolution of 3D scenes. Notably, it is **the first generative 4D occupancy world model for autonomous driving**. The model employs a 4D scene tokenizer to create compact spatiotemporal representations from 4D occupancy inputs, facilitating the high-quality reconstruction of long-sequence occupancy videos. Additionally, OccSora incorporates a diffusion transformer to generate 4D occupancy conditioned on trajectory prompts. Extensive experiments on the nuScenes dataset demonstrate OccSora's ability to generate 16-second videos with realistic 3D layouts and temporal consistency, highlighting its capacity to comprehend dynamic driving scenes. As a result, OccSora holds promise as a simulation tool for autonomous driving decision-making.
weaknesses: 1. I don't see the necessity of using quantizers and a codebook in the tokenizer (line 200). Since the authors employ a diffusion-based model, typically paired with a continuous compression model (e.g., VAE), this differs from the auto-regressive models (e.g., OccWorld) where such techniques might be more fitting. Could the author clarify their reasoning or provide supporting empirical results?
2. The idea of compressing the latent spatiotemporal dimensions together is interesting. However, in the diffusion model, simply flattening the tokens results in temporal modeling inefficient. Why not adopt an additional temporal layer, as is common in video-based diffusion models? Some references that might be helpful include:
  - Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
  - Scaling Latent Video Diffusion Models to Large Datasets
  - VDT: General-purpose Video Diffusion Transformers via Mask Modeling
  - Latte: Latent Diffusion Transformer for Video Generation
3. The generation metric is not described in sufficient detail. Firstly, FID is designed to evaluate images, yet this paper generates occupancy videos. At the very least, the authors should consider using FVD. Additionally, no details or references explain how FID is adapted for occupancy data or how occupancy features are extracted from pre-trained networks, given that this is the first work to generate 4D occupancy.
4. I am confused by the two experimental setups in Section 4.3. For the "Trajectory Video Generation" experiment (line 406), I expected to see how the generation would vary with different trajectory inputs. However, Figure 7 presents three entirely different scenes, even in the first frame. In the "Scene Video Generation" experiment (line 413), the authors claim they use the same trajectory for each motion case (line 417), but what causes the differences in the left and right parts of Figure 8? Is it the random seed or the input latents? The inference process is not explained at all.
5. I have concerns about the 4D occupancy prediction experiment described in line 484. The task is to forecast future frames based on historical frames. However, the authors use a 3D encoder with no masking strategy, which means the encoder can "see" the future frames, making the prediction task unfair. This could be proved by the results in Table 8 of the appendix, where the reconstruction IoU is higher for the 1-9s but lower at 0s, since there is no historical information for the 0s frame. Furthermore, the paper only demonstrates the process starting from pure noise. How is historical occupancy input into the DIT model?

Meta-review: This paper proposed a diffusion-based 4D occupancy generation model, OccSora, for autonomous driving. Essentially, the proposed method is a trajectory conditioned occupancy generation model. I summarize strengths and weaknesses as follows. 

Strengths:
1. The proposed world-model simulator task seems novel in autonomous driving. 
2. Extensive results on nuScenes demonstrate consistent occupancy prediction. 

Weaknesses:
1. Despite the overall claim seems novel, the techniques (diffusion for occupancy) are extensively studied in other works(e.g. DynamicCity, Pyramid Diffusion).
2. Comparisons with existing methods (e.g., OccWorld) are limited and, in some cases, favor OccSora due to differences in input and evaluation setups.
3. The paper claims a world simulator/model, however, only does occupancy prediction. It fails to provide adequate evidences of using this world simulator. 

Therefore, despite several merits, the paper still fails to meet the standard of ICLR due to technical novelty, rigorous comparisons, and adequate empirical justification of the proposed method.

Decision: Reject