Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving

Authors: Xiang Li, Pengfei Li, Yupeng Zheng, Wei Sun, Yan Wang, yilun chen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Autonomous Driving; Occupancy; World Model

Abstract:
Understanding world dynamics is crucial for planning in autonomous driving. Recent methods attempt to achieve this by learning a 3D occupancy world model that forecasts future surrounding scenes based on current observation. However, 3D occupancy labels are still required to produce promising results. Considering the high annotation cost for 3D outdoor scenes, we propose a semi-supervised vision-centric 3D occupancy world model, **PreWorld**, to leverage the potential of 2D labels through a novel two-stage training paradigm: the self-supervised pre-training stage and the fully-supervised fine-tuning stage. Specifically, during the pre-training stage, we utilize an attribute projection head to generate different attribute fields of a scene (e.g., RGB, density, semantic), thus enabling temporal supervision from 2D labels via volume rendering techniques. Furthermore, we introduce a simple yet effective state-conditioned forecasting module to recursively forecast future occupancy and ego trajectory in a direct manner. Extensive experiments on the nuScenes dataset validate the effectiveness and scalability of our method, and demonstrate that PreWorld achieves competitive performance across 3D occupancy prediction, 4D occupancy forecasting and motion planning tasks.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730383347120

thread_type:
official_review

<thread_object>
creator: reviewer_bypk

<content>
summary:
The paper proposes PreWorld, a semi-supervised, vision-centric 3D occupancy model that leverages 2D labels through a novel two-stage training approach. In the self-supervised pre-training stage, an attribute projection head is used to enable temporal supervision from 2D labels, while the fully-supervised fine-tuning stage introduces a state-conditioned forecasting module to directly predict future occupancy and ego trajectory.

strengths:
The paper presents a comprehensive approach, from the overall architectural design to thorough experimental validation:

a). The paper is well-written and easy to follow, presenting a method that is both well-motivated and clearly explained.

b). The proposed state-conditioned forecasting module is interesting.

weaknesses:
a). Although PreWorld leverages 2D labels to reduce the cost of 3D annotations, the paper only utilizes the nuScenes dataset for training, which does not fully demonstrate the method’s advantages. The semi-supervised approach should be highlighted by showcasing its benefits in terms of data efficiency.

b). Different tasks in occupancy predictions (4D occupancy forecasting, motion planning...) may have interdependent effects, where optimizing one module could potentially negatively impact others. The paper should provide a clear discussion on how to achieve a balance among these tasks and address potential interactions between the modules.

c). PreWorld’s two-stage training process may increase model complexity. This could lead to higher computational demands, potentially affecting the feasibility of real-time application in autonomous driving. The paper should further validate the feasibility of this method in terms of real-time inference performance.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730455497883

thread_type:
official_review

<thread_object>
creator: reviewer_fn92

<content>
summary:
The paper propose a semi-supervised method called Pre-World for occupancy forecasting problem. Pre-world first train the occupancy network in a self-supervised manner with the pre-training goal to render future image observation. Then Pre-world train the occupancy network with temporal occupancy labels to further improve the performance. Experiment results on NuScenes dataset show that the proposed method to some extend improve the performance compared to training from scratch.

strengths:
The paper is easy to follow. The proposed Preworld is able to utilize unlabeled data for pre-training and improve the performance of the downstream tasks. The experiments are extensive.

weaknesses:
1) The performance improvement seems not that significant. Generally, there are two kinds of improvement that pre-training can bring. The first one is to accelerate convergence in the downstream task. The second one is to improve sample efficiency in the downstream task, that is to improve the performance at convergence. The reviewer are concerned about whether the improvement presented in the experiment part is related to the acceleration of convergence.

2) It can be found in Table 1 that after pre-training, ious of some categories like driving surface and manmade drop. Also, compared to previous method, performance on some categories including manmade and vegetation are lower. But there is no analysis about these.

questions:
1)  Is it possible for the authors to provide experiments on training longer for both randomly initialization and pre-trained ones until they both converge in the downstream task?

2) Could the authors provide analysis on the detailed category performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582193134

thread_type:
official_review

<thread_object>
creator: reviewer_gcmx

<content>
summary:
The paper presents "PreWorld," a semi-supervised model aimed at improving 3D occupancy prediction and 4D forecasting for autonomous driving. It leverages both 2D and 3D data in a two-stage training approach, including a self-supervised pre-training phase using 2D labels and a fully-supervised fine-tuning phase with 3D labels. The authors introduce a state-conditioned forecasting module that directly predicts future occupancy and vehicle trajectory. Tests on the Occ3D-nuScenes dataset show PreWorld achieves competitive results.

strengths:
The paper provides a thorough evaluation of different methods on the Occ3D-nuScenes dataset. The experiments shows that the proposed model and training method can achieve competitive results across 3D occupancy prediction, 4D forecasting, and motion planning tasks.

weaknesses:
1. The contributions are largely incremental. While the state-conditioned forecasting module (Section 3.2) enables volume feature preservation for rendering, this approach aligns closely with existing practices in similar tasks​ [1][2]. The self-supervision in Section 3.3 extends volume rendering to 4D but follows typical methodologies in related work​ [3].
2. The figures are difficult to interpret due to insufficient captions. For instance, Figures 2 and 3 lack details, making it unclear how the components operate or interact. The difference between solid and dashed arrows in Figures 1, 2, and 3 is not explained. Figure 2 leaves confusion about whether the two stages are jointly or sequentially trained.
3. The authors claim 2D rendering as self-supervision, but the best result comes from utilizing depth and semantic labels, which is not self-supervision, and invalidates "semi-supervision" in the title.
4. The performance improvement is minimal. For example, the mIoU improvment for 3d occupancy prediction is only about 2% compared to OccFlowNet. And there is no significant improvement in other experiments and metrics

[1] Khurana, et al. "Point cloud forecasting as a proxy for 4d occupancy forecasting." CVPR 2023.

[2] Liu, et al. "Lidar-based 4d occupancy completion and forecasting." IROS 2024.

[3] Yang, et al. "Emernerf: Emergent spatial-temporal scene decomposition via self-supervision."

questions:
The proposed two stages can be trained jointly, why train them in two stages? Can the authors provide any motivation on this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730816492223

thread_type:
official_review

<thread_object>
creator: reviewer_z3jn

<content>
summary:
This paper presents a semi-supervised vision-centric 3D occupancy world mdoel, PreWorld. It leverages 2D labels through two-stage training, self-supervised pretraining and supervised finetuning. In the pre-training stage, the method utilizes an attribute projection head to enable temporal supervision from 2D labels. The method also includes a state-conditioned forecasting module.

strengths:
1. The self-supervised pre-training is simple and intuitive. The method projects predicted volume using attribute projection head, then use future 2D frames as supervision signal. It is effective by upgrading training strategy without introducting extra modules for inference.
2. The experiments are extensive to show superior performance in occupancy prediction and forecasting tasks.
3. The paper writing is clear, it is easy to read and follow.

weaknesses:
1. The improvement with and with-out pre-training seems not very significant. The module sometimes hurt performance (In tab.1). 
2. The ego state-conditioned module lacks of sufficient analysis and comparison.

questions:
1. "Our Copy&Paste"  in Tab.6 is confusing for me. What is the setting of row 2 in Tab. 6 and what is difference between row 2 and copy & paste?
2. How to correctly interrept the result in Tab.3? The method can achieve a comparable result even without ego-state information? Can you elaborate more about the result for ours with ego-state information.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732077672069

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you for your time and insightful feedback.

comment:
We would like to express our gratitude to the reviewers for acknowledging the strengths of our work, including its intuitiveness (**Reviewer z3jn**, **Reviewer BYPK**), clear writing (**Reviewer z3jn**, **Reviewer fN92**, **Reviewer BYPK**), extensive experiments (**Reviewer z3jn**, **Reviewer GcMx**, **Reviewer fN92**, **Reviewer BYPK**), and superior performance (**Reviewer z3jn**, **Reviewer GcMx**).

In response to the reviewers' questions, we have provided corresponding explanations and conducted further experiments. We are looking forward to further discussions!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732588090638

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank the reviewers for their insightful feedback, which we believe helped to significantly improve the manuscript.

We have included the responses to each reviewer’s comments and highlighted the changes in our revised manuscript in blue. Again, we really appreciate the constructive remarks and thank you all for your detailed feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces PreWorld, a semi-supervised, vision-centric 3D occupancy model for autonomous driving. It employs a two-stage training framework that integrates 2D labels during a self-supervised pretraining phase and 3D occupancy data during a supervised fine-tuning phase. The work aims to improve 3D occupancy prediction and 4D forecasting while maintaining scalability and leveraging cost-effective data annotation. In summary, reviewers believe this paper exhibits a novel training paradigm and complete experiments. Weaknesses include the incremental contributions, some initial concerns on the evaluation, and paper presentation. However, after the rebuttal, reviewers are all positive about this paper and concerns are (mostly) addressed. Therefore, I recommend accepting this paper.