OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving

Authors: Lening Wang, Wenzhao Zheng, Yilong Ren, Han Jiang, Zhiyong Cui, Haiyang Yu, Jiwen Lu
Primary area: applications to robotics, autonomy, planning
Keywords: Autonomous driving, world model

Abstract:
Understanding the evolution of 3D scenes is important for effective autonomous driving. While conventional methods model the scene development with the motion of individual instances, world models emerge as a generative framework to describe the general scene dynamics. However, most existing methods adopt an autoregressive framework to perform next-token prediction, which suffer from inefficiency to model long-term temporal evolutions.To address this, we propose a diffusion-based 4D occupancy generation model, OccSora, to simulate the development of the 3D world for autonomous driving. We employ a 4D scene tokenizer to obtain compact discrete spatial-temporal representations for 4D occupancy input and achieve high-quality reconstruction for long-sequence occupancy videos. We then learn a diffusion transformer on the spatial-temporal representations and generate 4D occupancy conditioned on a trajectory prompt. We conduct extensive experiments on the widely used nuScenes dataset with Occ3D occupancy annotations. OccSora can generate 16s videos with authentic 3D layout and temporal consistency, demonstrating its ability to understand the spatial and temporal distributions of driving scenes. With trajectory-aware 4D generation, OccSora has the potential to serve as a world simulator for the decision-making of autonomous driving.

Decision: reject
Reviewer scores: 5, 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729697625715

thread_type:
official_review

<thread_object>
creator: reviewer_wbnq

<content>
summary:
This paper introduces **OccSora**, a 4D occupancy generation model designed to capture the evolution of 3D scenes. Notably, it is **the first generative 4D occupancy world model for autonomous driving**. The model employs a 4D scene tokenizer to create compact spatiotemporal representations from 4D occupancy inputs, facilitating the high-quality reconstruction of long-sequence occupancy videos. Additionally, OccSora incorporates a diffusion transformer to generate 4D occupancy conditioned on trajectory prompts. Extensive experiments on the nuScenes dataset demonstrate OccSora's ability to generate 16-second videos with realistic 3D layouts and temporal consistency, highlighting its capacity to comprehend dynamic driving scenes. As a result, OccSora holds promise as a simulation tool for autonomous driving decision-making.

strengths:
1. The paper is well-organized, making it easy to follow.
2. This paper introduces the first generative 4D occupancy world model for autonomous driving and proposes a new generation task for occupancy data, which is a contribution to the community.
3. The design of the 4D tokenizer (Section 3.2) is particularly novel. The experiments demonstrate its effectiveness in preserving 3D occupancy geometry with temporal consistency, even under high compression rates.
4. The ablation study is thorough and comprehensive.

weaknesses:
1. I don't see the necessity of using quantizers and a codebook in the tokenizer (line 200). Since the authors employ a diffusion-based model, typically paired with a continuous compression model (e.g., VAE), this differs from the auto-regressive models (e.g., OccWorld) where such techniques might be more fitting. Could the author clarify their reasoning or provide supporting empirical results?
2. The idea of compressing the latent spatiotemporal dimensions together is interesting. However, in the diffusion model, simply flattening the tokens results in temporal modeling inefficient. Why not adopt an additional temporal layer, as is common in video-based diffusion models? Some references that might be helpful include:
  - Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
  - Scaling Latent Video Diffusion Models to Large Datasets
  - VDT: General-purpose Video Diffusion Transformers via Mask Modeling
  - Latte: Latent Diffusion Transformer for Video Generation
3. The generation metric is not described in sufficient detail. Firstly, FID is designed to evaluate images, yet this paper generates occupancy videos. At the very least, the authors should consider using FVD. Additionally, no details or references explain how FID is adapted for occupancy data or how occupancy features are extracted from pre-trained networks, given that this is the first work to generate 4D occupancy.
4. I am confused by the two experimental setups in Section 4.3. For the "Trajectory Video Generation" experiment (line 406), I expected to see how the generation would vary with different trajectory inputs. However, Figure 7 presents three entirely different scenes, even in the first frame. In the "Scene Video Generation" experiment (line 413), the authors claim they use the same trajectory for each motion case (line 417), but what causes the differences in the left and right parts of Figure 8? Is it the random seed or the input latents? The inference process is not explained at all.
5. I have concerns about the 4D occupancy prediction experiment described in line 484. The task is to forecast future frames based on historical frames. However, the authors use a 3D encoder with no masking strategy, which means the encoder can "see" the future frames, making the prediction task unfair. This could be proved by the results in Table 8 of the appendix, where the reconstruction IoU is higher for the 1-9s but lower at 0s, since there is no historical information for the 0s frame. Furthermore, the paper only demonstrates the process starting from pure noise. How is historical occupancy input into the DIT model?

questions:
1. **The main text of the paper exceeds the maximum page limit of 10 pages**, as per the ICLR 2025 guidelines. Please check the details at ICLR Call for Papers.
2. The authors should consider using `\citep{}` in the paper for better formatting. Refer to the ICLR LaTeX template for the appropriate usage and differences between citation commands.
3. In the caption of Table 1, the output dimension for the "512x" setting should be listed as $ 25 \times 25 \times 4 $.
4. In Figure 6, the model name should be updated to **OccSora-base**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730124550527

thread_type:
official_review

<thread_object>
creator: reviewer_e9cj

<content>
summary:
This paper proposes OccSora, a diffusion-based 4D occupancy generation model designed for autonomous driving. It utilizes a 4D scene tokenizer to get spatial-temporal representations, allowing for high-quality reconstruction of long-term occupancy videos. The model generates 4D occupancy data based on trajectory prompts, generating videos with 3D layouts and temporal consistency. Experiments demonstrate OccSora's potential as a world simulator for autonomous driving.

strengths:
1. This paper is well-organized, presenting a clear flow making it easy to read.
2. This paper proposes a generative 4D occupancy world model designed for autonomous driving, along with a novel generation task for occupancy data.
3. The experiments provide an evaluation of the model's capabilities and performance across various scenarios.

weaknesses:
1. The quality of the turning cases shown in Figure 8 and Figure 13 appears to be quite poor. The motion trend is barely visible, and the scene structure beside the road becomes corrupted in the later frames. This undermines the claim of trajectory awareness. I suspect this issue is related to the imbalanced data distribution in the nuScenes dataset, which could present an interesting challenge. Unfortunately, the paper offers no further analysis or solution regarding this problem.
2. The authors spent a lot of space discussing the diffusion transformer (lines 269–290). However, much of this appears to replicate the contribution of the original DIT paper. This section could be shortened or moved to a preliminary section instead.
3. The mathematical notation throughout the paper is quite disorganized. Some of the symbols are unusual, and the authors should consider using more standard notation (e.g., what is "mi" in $ R_{mi} $?). The excessive use of superscripts and subscripts makes it difficult to follow. Additionally, the authors should avoid using unnecessary symbols for unimportant variables (e.g., $ x $ in line 246). There are also several instances of misused symbols. For example:
  1. The symbol $ N $ is used multiple times for different meanings, such as a label (line 176), nearest code operation (line 205), and Gaussian noise (line 271).
  2. In line 254, the positional encoding notation is inconsistent with Equation (2).
  3. In Equation (3), the $ t $ in $ v(t) $ should represent the diffusion timestep, but it is confused with the occupancy timestep $ t $.
  4. In line 265, $ v $ is referred to as the waypoint timestep embedding, but it is suddenly called the denoising timestep in line 269.

4. There are several grammar issues and typos:
  1. In line 181, "represents" should be "representing."
  2. In line 197, "model ability" should be "model's ability."
  3. In line 253, "model understanding" should be "model's understanding."
  4. In line 266, it should read: " $ g $ is then embedded into the input sequence ..."
  5. In line 280, "occuancy" should be "occupancy."
  6. In line 525, "reprort" should be "report."

questions:
1. In Table 2, it's unclear why increasing the resolution (from $ 128 \times 4 \times 25 \times 25 $ to $ 128 \times 8 \times 50 \times 50 $) improves the reconstruction IoU and mIoU, but results in a decrease in generation FID. This discrepancy needs further explanation.
2. The figures showing occupancy data throughout the paper are quite small, and the resolution is low, making it difficult to see details. The authors should consider placing fewer samples in each row and enlarging the figures for better visibility.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730377177403

thread_type:
official_review

<thread_object>
creator: reviewer_y4yg

<content>
summary:
The paper introduces a trajectory-aware 4D occupancy generation model capable of understanding ego car trajectories and enabling trajectory-controllable scene generation.

strengths:
This paper proposes a trajectory-aware 4D occupancy generation model, which can comprehend the trajectories of ego car and realize trajectory-controllable scene generation. The model uses a 4D scene tokenizer to create compact, discrete spatial-temporal representations of 4D occupancy inputs, achieving high-quality reconstructions for long-sequence occupancy videos. The proposed OccSora can generate long-term occupancy videos up to 16 seconds.

weaknesses:
1. This paper claims in the abstract and introduction that existing autoregressive methods "suffer from inefficiency to model long-term temporal evolutions". It is unclear how the authors address this, as the multiple denoising steps of the diffusion model can also be time-consuming. I suggest that the authors provide some **evidences to demonstrate that the proposed architecture is more efficient**, such as runtime comparisons or theoretical complexity analyses between the proposed approach and existing autoregressive methods.
2. The experiment on the 4D occupancy prediction is not convincing. The metrics indicate that OccSora significantly outperforms OccWorld; however, the **generation quality** in qualitative results is not good enough. It is beneficial to provide side-by-side visual comparisons between OccSora and baseline methods, highlighting specific areas where the generation quality differs.
3. It is unclear **how 4D occupancy prediction is conducted** in Fig. 5 and Tab. 10, as there appears to be no frame condition in the proposed architecture. I suggest that the authors provide a more detailed explanation of how the model handles frame conditioning for 4D occupancy prediction in the proposed architecture.
4. As shown in Fig.12, the model can generate different scenes with different motion conditions. However, their first frames are different. As far as the reviewer knows, if the authors want to **claim the model is controllable**, the model should be able to generate different future predictions with the same initial frame(s) using different trajectories. I recommend that the authors provide additional experiments or qualitative results to demonstrate this .
5. Why are the IoU and mIoU values at 0s in Table 5 different from the reconstruction results in Table 1? Both seem to measure reconstruction performance.

questions:
The reviewer has identified five major concerns and would like the authors' responses to these points. Please answer each concern in the rebuttal stage. The reviewer will respond according to the authors' rebuttal in the discussion phase.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710236380

thread_type:
official_review

<thread_object>
creator: reviewer_t69g

<content>
summary:
This paper introduces OccSora, a diffusion-based model for 4D occupancy generation in autonomous driving. Using a 4D scene tokenizer, OccSora captures compact and discrete spatiotemporal representations, allowing for the high-quality reconstruction of long-sequence occupancy videos. A diffusion transformer is subsequently trained on these representations to generate 4D occupancy, conditioned on a trajectory prompt. Experimental results on the nuScenes dataset with Occ3D occupancy annotations show that OccSora can produce 16-second videos with realistic 3D layouts and temporal coherence, demonstrating its understanding of spatial and temporal patterns in driving scenes. This method shows potential as a world simulator for decision-making in autonomous driving, addressing the inefficiencies in modeling long-term temporal evolution found in autoregressive approaches.

strengths:
This paper presents OccSora, a pioneering diffusion-based 4D occupancy generation model that utilizes a 4D scene tokenizer to capture compact spatiotemporal representations, improving the modeling of long-term temporal dynamics in autonomous driving. The research is comprehensive, featuring extensive experiments on the nuScenes dataset that showcase OccSora's ability to generate realistic 16-second videos with stable 3D layouts.  The paper is clearly structured and well-written, offering thorough explanations of the model architecture, training procedure, and evaluation metrics. OccSora’s capability to generate trajectory-aware 4D occupancy scenes makes it a valuable asset for decision-making in autonomous driving, with the potential to enhance safety and efficiency.

weaknesses:
The writing of this paper should be enhanced. For example,

1）Some details in the methods section (Section 3) could be presented with greater clarity. For ‘Category embedding and Tokenizer’, occupancy is represented by Rin in the text, while in Figure 3, occupancy is represented by Ro.

2）If Figures 2, Figure 3 and Figure 4 contain elements corresponding to Rin, Rmi, Ro, and Rtr, please ensure these are labeled directly within the figures.

3）Please clearly indicate the dimensional changes between features within the figure to enhance reader comprehension.

questions:
Refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730846393371

thread_type:
official_review

<thread_object>
creator: reviewer_cjwx

<content>
summary:
The paper introduces a 4D occupancy generation model called OccSora, designed to simulate the evolution of environments in autonomous driving scenarios. It proposes a 4D scene encoder to produce a compact spatiotemporal representation and uses a diffusion model to generate 4D occupancy sequences based on given trajectory prompts. OccSora addresses the inefficiency of existing autoregressive models in long-term temporal simulation, providing more physically consistent simulation support for decision-making in autonomous driving through trajectory-aware 4D generation. OccSora achieves good results on nuScenes dataset.

strengths:
1.The paper is well written and organized. It’s easy to understand.
2.The authors provide abundant visualization results to show the effectiveness.

weaknesses:
1. The novelty is limited. Using diffusion to generate occupancy is widely studied. And the contribution seems incremental compared to previous works in the field of occupancy prediction.
2.  It might be beneficial to conduct experiments on additional datasets, such as Waymo, to enhance the persuasiveness of the paper. 
3. The text in Figure 3 is too small and unclear.

questions:
See the weaknesses above
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a diffusion-based 4D occupancy generation model, OccSora, for autonomous driving. Essentially, the proposed method is a trajectory conditioned occupancy generation model. I summarize strengths and weaknesses as follows. 

Strengths:
1. The proposed world-model simulator task seems novel in autonomous driving. 
2. Extensive results on nuScenes demonstrate consistent occupancy prediction. 

Weaknesses:
1. Despite the overall claim seems novel, the techniques (diffusion for occupancy) are extensively studied in other works(e.g. DynamicCity, Pyramid Diffusion).
2. Comparisons with existing methods (e.g., OccWorld) are limited and, in some cases, favor OccSora due to differences in input and evaluation setups.
3. The paper claims a world simulator/model, however, only does occupancy prediction. It fails to provide adequate evidences of using this world simulator. 

Therefore, despite several merits, the paper still fails to meet the standard of ICLR due to technical novelty, rigorous comparisons, and adequate empirical justification of the proposed method.